Nov 02 17:26:53.194549 ip-10-0-136-153 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 17:26:53.194565 ip-10-0-136-153 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 17:26:53.194574 ip-10-0-136-153 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 17:26:53.194928 ip-10-0-136-153 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 17:27:03.328954 ip-10-0-136-153 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 17:27:03.328970 ip-10-0-136-153 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 2804984d2b9b469893b19653375ab2ab -- Nov 02 17:28:59.946174 ip-10-0-136-153 systemd[1]: Starting Kubernetes Kubelet... Nov 02 17:29:00.338637 ip-10-0-136-153 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:29:00.338637 ip-10-0-136-153 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 17:29:00.338637 ip-10-0-136-153 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:29:00.338637 ip-10-0-136-153 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:29:00.343357 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343276 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:00.343357 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343358 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:00.343431 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343364 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:00.343431 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343367 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:00.343431 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343370 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:00.343431 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343373 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:00.343431 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343376 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:00.343431 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343384 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:00.343431 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343388 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:00.343431 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343391 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:00.343431 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343393 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:00.343431 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343396 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:00.343431 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343399 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:00.343431 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343401 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:00.343431 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343404 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:00.343431 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343407 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:00.343431 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343409 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:00.343431 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343412 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:00.343431 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343414 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:00.343431 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343417 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:00.343431 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343419 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:00.343874 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343422 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:00.343874 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343424 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:00.343874 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343427 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:00.343874 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343430 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:00.343874 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343432 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:00.343874 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343435 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:00.343874 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343438 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:00.343874 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343441 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:00.343874 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343444 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:00.343874 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343447 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:00.343874 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343449 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:00.343874 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343452 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:00.343874 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343454 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:00.343874 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343457 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:00.343874 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343459 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:00.343874 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343462 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:00.343874 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343464 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:00.343874 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343467 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:00.343874 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343470 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:00.344317 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343473 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:00.344317 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343475 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:00.344317 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343478 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:00.344317 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343480 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:00.344317 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343483 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:00.344317 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343485 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:00.344317 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343488 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:00.344317 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343490 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:00.344317 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343493 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:00.344317 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343495 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:00.344317 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343498 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:00.344317 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343500 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:00.344317 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343503 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:00.344317 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343505 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:00.344317 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343508 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:00.344317 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343511 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:00.344317 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343515 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:00.344317 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343519 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:00.344317 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343522 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343525 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343528 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343531 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343533 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343536 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343539 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343542 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343545 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343548 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343551 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343553 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343556 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343559 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343561 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343564 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343568 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343572 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343575 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343578 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:00.344768 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343581 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343584 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343587 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343589 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343592 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343595 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343598 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343601 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343605 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343608 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343611 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343613 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343616 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343618 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343621 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343624 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343626 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343629 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343632 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343634 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:00.345543 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343637 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343639 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343642 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343645 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343648 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343651 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343653 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.343656 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344600 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344609 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344613 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344615 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344618 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344621 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344624 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344626 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344629 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344632 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344634 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344637 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:00.346251 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344640 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:00.346724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344643 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:00.346724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344645 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:00.346724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344648 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:00.346724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344651 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:00.346724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344654 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:00.346724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344656 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:00.346724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344659 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:00.346724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344661 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:00.346724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344664 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:00.346724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344667 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:00.346724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344669 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:00.346724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344672 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:00.346724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344675 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:00.346724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344679 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:00.346724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344683 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:00.346724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344686 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:00.346724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344688 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:00.346724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344691 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:00.346724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344694 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:00.347196 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344697 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:00.347196 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344700 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:00.347196 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344702 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:00.347196 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344705 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:00.347196 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344707 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:00.347196 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344710 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:00.347196 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344712 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:00.347196 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344715 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:00.347196 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344718 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:00.347196 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344721 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:00.347196 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344724 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:00.347196 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344727 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:00.347196 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344730 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:00.347196 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344732 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:00.347196 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344735 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:00.347196 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344738 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:00.347196 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344740 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:00.347196 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344743 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:00.347196 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344745 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344750 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344753 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344757 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344760 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344763 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344765 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344768 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344771 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344774 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344777 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344779 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344782 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344784 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344827 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344919 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344925 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344929 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344934 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344938 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:00.347642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344943 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:00.348130 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344947 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:00.348130 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344952 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:00.348130 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344956 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:00.348130 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344961 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:00.348130 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344966 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:00.348130 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344971 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:00.348130 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344976 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:00.348130 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344985 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:00.348130 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344989 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:00.348130 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344994 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:00.348130 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.344998 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:00.348130 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345002 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:00.348130 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345007 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:00.348130 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345012 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:00.348130 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345016 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:00.348130 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345021 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:00.348130 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345025 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:00.348130 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345029 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:00.348130 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345034 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:00.348608 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345038 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:00.348608 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345047 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:00.348608 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345051 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:00.348608 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345056 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:00.348608 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345061 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:00.348608 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345065 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:00.348608 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345070 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:00.348608 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345095 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:00.348608 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345099 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:00.348608 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345105 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:00.348608 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345110 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:00.348608 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345114 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:00.348608 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345118 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:00.348608 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345129 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:00.348608 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345135 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:00.348608 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.345139 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:00.348608 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346402 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:00.348608 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346412 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:00.348608 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346417 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:00.349081 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346420 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:00.349081 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346423 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:00.349081 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346426 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:00.349081 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346431 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:00.349081 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346434 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:00.349081 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346437 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:00.349081 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346440 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:00.349081 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346443 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:00.349081 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346446 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:00.349081 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346449 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:00.349081 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346451 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:00.349081 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346454 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:00.349081 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346456 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:00.349081 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346459 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:00.349081 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346462 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:00.349081 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346465 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:00.349081 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346467 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:00.349081 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346470 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:00.349081 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346473 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346475 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346478 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346481 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346483 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346486 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346489 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346492 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346494 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346497 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346499 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346502 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346505 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346508 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346511 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346514 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346516 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346519 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346522 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346524 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:00.349517 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346526 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346529 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346532 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346534 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346537 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346539 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346542 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346545 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346547 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346550 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346553 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346556 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346559 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346561 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346564 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346566 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346569 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346572 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346574 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346577 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:00.350000 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346579 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:00.350570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346582 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:00.350570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346584 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:00.350570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346587 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:00.350570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346592 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:00.350570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346594 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:00.350570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346597 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:00.350570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346599 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:00.350570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346602 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:00.350570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346605 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:00.350570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346607 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:00.350570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346610 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:00.350570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346612 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:00.350570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346615 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:00.350570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346618 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:00.350570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346620 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:00.350570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346623 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:00.350570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346625 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:00.350570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346628 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:00.350570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346631 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:00.351066 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346633 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:00.351066 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346638 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:00.351066 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346643 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:00.351066 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346646 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:00.351066 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346649 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:00.351066 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346652 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:00.351066 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346655 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:00.351066 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346657 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:00.351066 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346660 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:00.351066 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346663 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:00.351066 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346665 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:00.351066 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346668 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:00.351066 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346670 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:00.351066 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346673 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:00.351066 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346675 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:00.351066 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346678 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:00.351066 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346682 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:00.351066 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346685 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:00.351066 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346688 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346690 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346693 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346697 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346700 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346702 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.346705 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348183 2576 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348193 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348200 2576 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348205 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348209 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348213 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348217 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348221 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348224 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348228 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348232 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348235 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348238 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348241 2576 flags.go:64] FLAG: --cgroup-root="" Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348244 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348248 2576 flags.go:64] FLAG: --client-ca-file="" Nov 02 17:29:00.351506 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348251 2576 flags.go:64] FLAG: --cloud-provider="external" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348254 2576 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348259 2576 flags.go:64] FLAG: --cluster-domain="" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348262 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348265 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348268 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348272 2576 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348277 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348280 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348283 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348286 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348289 2576 flags.go:64] FLAG: --contention-profiling="false" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348292 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348295 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348298 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348301 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348305 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348309 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348312 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348315 2576 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348318 2576 flags.go:64] FLAG: --enable-server="true" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348321 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348325 2576 flags.go:64] FLAG: --event-burst="100" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348328 2576 flags.go:64] FLAG: --event-qps="50" Nov 02 17:29:00.352051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348331 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348334 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348337 2576 flags.go:64] FLAG: --eviction-hard="" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348342 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348345 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348348 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348351 2576 flags.go:64] FLAG: --eviction-soft="" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348354 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348357 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348360 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348363 2576 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348366 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348370 2576 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348372 2576 flags.go:64] FLAG: --feature-gates="" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348389 2576 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348393 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348397 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348400 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348404 2576 flags.go:64] FLAG: --healthz-port="10248" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348407 2576 flags.go:64] FLAG: --help="false" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348410 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-136-153.ec2.internal" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348413 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348416 2576 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 17:29:00.352639 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348419 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348422 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348426 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348429 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348432 2576 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348436 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348439 2576 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348442 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348445 2576 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348448 2576 flags.go:64] FLAG: --kube-reserved="" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348451 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348454 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348457 2576 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348460 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348463 2576 flags.go:64] FLAG: --lock-file="" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348466 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348469 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348472 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348478 2576 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348481 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348484 2576 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348487 2576 flags.go:64] FLAG: --logging-format="text" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348490 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348493 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 17:29:00.353185 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348496 2576 flags.go:64] FLAG: --manifest-url="" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348499 2576 flags.go:64] FLAG: --manifest-url-header="" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348504 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348507 2576 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348511 2576 flags.go:64] FLAG: --max-pods="110" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348515 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348518 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348521 2576 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348524 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348527 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348530 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348533 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348540 2576 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348544 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348547 2576 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348550 2576 flags.go:64] FLAG: --pod-cidr="" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348553 2576 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348556 2576 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348559 2576 flags.go:64] FLAG: --pods-per-core="0" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348562 2576 flags.go:64] FLAG: --port="10250" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348565 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348568 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dc428c8daadfdbe1" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348571 2576 flags.go:64] FLAG: --qos-reserved="" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348574 2576 flags.go:64] FLAG: --read-only-port="10255" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348577 2576 flags.go:64] FLAG: --register-node="true" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348580 2576 flags.go:64] FLAG: --register-with-taints="" Nov 02 17:29:00.353745 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348583 2576 flags.go:64] FLAG: --registry-burst="10" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348586 2576 flags.go:64] FLAG: --registry-qps="5" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348589 2576 flags.go:64] FLAG: --reserved-cpus="" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348592 2576 flags.go:64] FLAG: --reserved-memory="" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348596 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348599 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348602 2576 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348605 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348609 2576 flags.go:64] FLAG: --runonce="false" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348612 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348615 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348619 2576 flags.go:64] FLAG: --seccomp-default="false" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348622 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348625 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348628 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348631 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348635 2576 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348638 2576 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348640 2576 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348644 2576 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348647 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348650 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348653 2576 flags.go:64] FLAG: --system-cgroups="" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348656 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348664 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 17:29:00.354448 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348667 2576 flags.go:64] FLAG: --tls-cert-file="" Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348670 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348674 2576 flags.go:64] FLAG: --tls-min-version="" Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348677 2576 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348680 2576 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348683 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348686 2576 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348689 2576 flags.go:64] FLAG: --v="2" Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348698 2576 flags.go:64] FLAG: --version="false" Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348704 2576 flags.go:64] FLAG: --vmodule="" Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348708 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.348711 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348845 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348861 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348864 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348866 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348869 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348872 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348875 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348878 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348881 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348884 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:00.355064 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348886 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348889 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348892 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348894 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348897 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348900 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348903 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348906 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348908 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348913 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348916 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348918 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348921 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348923 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348926 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348929 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348931 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348934 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348936 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348941 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:00.355591 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348943 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:00.356076 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348946 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:00.356076 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348949 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:00.356076 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348952 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:00.356076 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348956 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:00.356076 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348959 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:00.356076 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348962 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:00.356076 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348965 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:00.356076 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348967 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:00.356076 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348970 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:00.356076 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348973 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:00.356076 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348976 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:00.356076 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348978 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:00.356076 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348981 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:00.356076 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348983 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:00.356076 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348986 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:00.356076 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348988 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:00.356076 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348991 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:00.356076 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348994 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:00.356076 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348997 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:00.356532 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.348999 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:00.356532 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349003 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:00.356532 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349006 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:00.356532 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349009 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:00.356532 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349011 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:00.356532 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349014 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:00.356532 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349016 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:00.356532 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349019 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:00.356532 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349021 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:00.356532 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349024 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:00.356532 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349027 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:00.356532 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349030 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:00.356532 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349033 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:00.356532 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349036 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:00.356532 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349039 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:00.356532 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349042 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:00.356532 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349044 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:00.356532 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349047 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:00.356532 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349049 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:00.356996 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349052 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:00.356996 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349055 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:00.356996 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349058 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:00.356996 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349060 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:00.356996 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349063 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:00.356996 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349066 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:00.356996 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349068 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:00.356996 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349070 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:00.356996 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349073 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:00.356996 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349076 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:00.356996 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349078 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:00.356996 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349081 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:00.356996 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349084 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:00.356996 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349087 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:00.356996 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349091 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:00.356996 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349093 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:00.356996 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349096 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:00.356996 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349098 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:00.356996 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349101 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:00.357447 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349104 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:00.357447 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349108 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:00.357447 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349112 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:00.357447 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349115 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:00.357447 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349118 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:00.357447 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349123 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:00.357447 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349126 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:00.357447 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349129 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:00.357447 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349131 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:00.357447 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349134 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:00.357447 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349136 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:00.357447 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349139 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:00.357447 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349142 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:00.357447 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349144 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:00.357447 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349147 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:00.357447 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349149 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:00.357447 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349152 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:00.357447 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349154 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:00.357937 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.349181 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:29:00.357937 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349223 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:00.357937 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349228 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:00.357937 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349231 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:00.357937 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349233 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:00.357937 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349236 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:00.357937 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349238 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:00.357937 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349241 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:00.357937 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349245 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:00.357937 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349249 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:00.357937 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349252 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:00.357937 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349255 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:00.357937 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349257 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:00.357937 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349260 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:00.357937 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349263 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:00.357937 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349266 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:00.358310 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349268 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:00.358310 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349271 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:00.358310 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349273 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:00.358310 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349276 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:00.358310 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349280 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:00.358310 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349282 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:00.358310 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349285 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:00.358310 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349287 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:00.358310 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349290 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:00.358310 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349293 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:00.358310 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349295 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:00.358310 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349298 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:00.358310 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349301 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:00.358310 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349303 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:00.358310 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349306 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:00.358310 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349308 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:00.358310 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349311 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:00.358310 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349313 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:00.358310 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349317 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349321 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349324 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349328 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349331 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349333 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349336 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349341 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349344 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349346 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349349 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349351 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349354 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349357 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349359 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349362 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349364 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349367 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349370 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349373 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:00.358770 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349376 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349379 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349382 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349385 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349388 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349390 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349393 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349395 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349398 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349401 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349403 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349406 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349408 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349411 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349414 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349416 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349419 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349421 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349424 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349428 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:00.359253 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349431 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349434 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349436 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349439 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349442 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349445 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349447 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349450 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349452 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349455 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349459 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349463 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349466 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349470 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349472 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349475 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349478 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349481 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349484 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349487 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:00.359801 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349489 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:00.360281 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349492 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:00.360281 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349495 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:00.360281 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349497 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:00.360281 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349500 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:00.360281 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349502 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:00.360281 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349506 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:00.360281 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349509 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:00.360281 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349511 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:00.360281 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349514 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:00.360281 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349517 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:00.360281 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349520 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:00.360281 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.349523 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:00.360281 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.355937 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 17:29:00.360281 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.356066 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 17:29:00.360281 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.356082 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 17:29:00.360281 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.356087 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 17:29:00.360281 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356143 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:00.360281 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356148 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:00.360281 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356151 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:00.360737 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356154 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:00.360737 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356157 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:00.360737 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356160 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:00.360737 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356163 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:00.360737 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356166 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:00.360737 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356169 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:00.360737 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356171 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:00.360737 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356174 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:00.360737 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356178 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:00.360737 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356182 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:00.360737 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356185 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:00.360737 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356188 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:00.360737 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356191 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:00.360737 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356194 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:00.360737 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356197 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:00.360737 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356200 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:00.360737 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356202 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:00.360737 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356205 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:00.360737 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356208 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356210 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356213 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356216 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356218 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356221 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356223 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356227 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356231 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356234 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356237 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356239 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356243 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356245 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356248 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356251 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356253 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356256 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356258 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356261 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:00.361194 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356263 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:00.361661 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356266 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:00.361661 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356269 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:00.361661 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356271 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:00.361661 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356274 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:00.361661 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356276 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:00.361661 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356279 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:00.361661 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356282 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:00.361661 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356284 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:00.361661 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356287 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:00.361661 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356289 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:00.361661 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356292 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:00.361661 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356294 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:00.361661 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356297 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:00.361661 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356299 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:00.361661 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356302 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:00.361661 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356305 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:00.361661 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356308 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:00.361661 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356310 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:00.361661 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356314 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:00.362202 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356317 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:00.362202 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356320 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:00.362202 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356323 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:00.362202 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356326 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:00.362202 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356328 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:00.362202 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356331 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:00.362202 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356335 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:00.362202 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356339 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:00.362202 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356342 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:00.362202 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356345 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:00.362202 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356348 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:00.362202 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356350 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:00.362202 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356353 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:00.362202 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356355 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:00.362202 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356358 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:00.362202 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356361 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:00.362202 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356363 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:00.362202 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356366 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:00.362202 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356368 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356371 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356374 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356376 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356379 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356389 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356393 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356395 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356398 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356401 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356404 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356406 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356409 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356412 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356415 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356417 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356421 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356424 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356427 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356429 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:00.362642 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356432 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:00.363131 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356434 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:00.363131 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356437 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:00.363131 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356439 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:00.363131 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356441 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:00.363131 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356444 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:00.363131 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356447 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:00.363131 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.356475 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:29:00.363131 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356615 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:00.363131 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356620 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:00.363131 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356624 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:00.363131 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356627 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:00.363131 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356631 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:00.363131 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356634 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:00.363131 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356637 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:00.363131 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356640 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:00.363131 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356642 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:00.363499 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356645 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:00.363499 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356647 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:00.363499 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356650 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:00.363499 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356653 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:00.363499 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356655 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:00.363499 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356657 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:00.363499 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356660 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:00.363499 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356663 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:00.363499 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356666 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:00.363499 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356669 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:00.363499 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356671 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:00.363499 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356674 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:00.363499 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356677 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:00.363499 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356679 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:00.363499 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356682 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:00.363499 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356684 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:00.363499 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356687 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:00.363499 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356689 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:00.363499 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356692 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:00.363956 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356694 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:00.363956 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356697 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:00.363956 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356699 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:00.363956 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356702 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:00.363956 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356704 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:00.363956 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356707 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:00.363956 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356709 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:00.363956 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356711 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:00.363956 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356714 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:00.363956 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356717 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:00.363956 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356719 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:00.363956 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356722 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:00.363956 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356724 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:00.363956 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356727 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:00.363956 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356729 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:00.363956 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356732 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:00.363956 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356734 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:00.363956 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356737 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:00.363956 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356740 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:00.364390 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356743 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:00.364390 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356745 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:00.364390 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356748 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:00.364390 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356751 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:00.364390 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356753 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:00.364390 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356756 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:00.364390 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356759 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:00.364390 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356762 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:00.364390 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356764 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:00.364390 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356768 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:00.364390 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356772 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:00.364390 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356775 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:00.364390 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356779 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:00.364390 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356781 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:00.364390 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356784 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:00.364390 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356788 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:00.364390 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356790 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:00.364390 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356793 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:00.364390 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356795 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356798 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356800 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356803 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356807 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356810 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356813 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356815 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356818 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356820 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356823 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356825 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356828 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356830 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356833 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356836 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356839 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356841 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356844 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356860 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:00.364855 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356862 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356865 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356868 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356871 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356874 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356876 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356879 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356881 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356884 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356886 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356889 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356891 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356894 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356897 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356899 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356902 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356904 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356907 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356909 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356912 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:00.365338 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356915 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:00.366108 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.356938 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:29:00.366108 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356970 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:00.366108 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356974 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:00.366108 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356977 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:00.366108 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356980 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:00.366108 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356983 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:00.366108 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356986 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:00.366108 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356989 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:00.366108 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356992 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:00.366108 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356994 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:00.366108 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.356997 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:00.366108 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357000 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:00.366108 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357003 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:00.366108 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357005 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:00.366108 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357009 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:00.366108 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357011 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357014 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357017 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357019 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357022 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357024 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357027 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357030 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357032 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357035 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357037 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357040 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357042 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357045 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357047 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357050 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357052 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357055 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357058 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357060 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:00.366503 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357062 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:00.366984 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357065 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:00.366984 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357068 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:00.366984 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357073 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:00.366984 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357075 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:00.366984 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357078 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:00.366984 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357081 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:00.366984 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357083 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:00.366984 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357086 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:00.366984 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357089 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:00.366984 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357092 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:00.366984 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357094 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:00.366984 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357097 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:00.366984 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357100 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:00.366984 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357103 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:00.366984 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357105 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:00.366984 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357108 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:00.366984 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357110 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:00.366984 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357113 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:00.366984 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357116 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:00.367417 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357118 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:00.367417 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357121 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:00.367417 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357124 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:00.367417 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357126 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:00.367417 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357130 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:00.367417 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357134 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:00.367417 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357138 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:00.367417 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357141 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:00.367417 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357143 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:00.367417 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357146 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:00.367417 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357149 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:00.367417 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357151 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:00.367417 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357154 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:00.367417 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357156 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:00.367417 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357159 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:00.367417 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357163 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:00.367417 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357166 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:00.367417 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357170 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:00.367417 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357173 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357176 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357179 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357182 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357184 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357187 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357189 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357195 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357198 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357201 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357203 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357206 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357208 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357211 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357214 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357217 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357219 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357222 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357224 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357227 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:00.367904 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357229 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:00.368361 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357232 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:00.368361 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357234 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:00.368361 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357237 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:00.368361 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357239 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:00.368361 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357242 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:00.368361 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357244 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:00.368361 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357247 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:00.368361 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357249 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:00.368361 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357253 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:00.368361 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357256 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:00.368361 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357258 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:00.368361 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357261 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:00.368361 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:00.357263 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:00.368361 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.357321 2576 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 17:29:00.368361 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.359321 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 17:29:00.368361 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.360206 2576 server.go:1014] "Starting client certificate rotation" Nov 02 17:29:00.368361 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.360300 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:29:00.368361 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.360338 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:29:00.368793 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.362157 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:29:00.368793 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.365592 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:29:00.376621 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.376602 2576 log.go:25] "Validated CRI v1 runtime API" Nov 02 17:29:00.379194 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.379180 2576 log.go:25] "Validated CRI v1 image API" Nov 02 17:29:00.380034 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.380013 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 17:29:00.385428 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.385409 2576 fs.go:135] Filesystem UUIDs: map[5924b1c7-711f-43f8-ab68-87abd407560c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f988989c-f35d-46ed-b30f-7d7057b62aa5:/dev/nvme0n1p3] Nov 02 17:29:00.385541 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.385427 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 17:29:00.390654 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.390548 2576 manager.go:217] Machine: {Timestamp:2026-11-02 17:29:00.389247461 +0000 UTC m=+0.327728671 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097793 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22aac5984598ac859f168d3fa2630c SystemUUID:ec22aac5-9845-98ac-859f-168d3fa2630c BootID:2804984d-2b9b-4698-93b1-9653375ab2ab Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d8:2a:eb:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d8:2a:eb:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:d2:fa:56:ac:72 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 17:29:00.390728 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.390655 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 17:29:00.390777 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.390766 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 17:29:00.394465 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.394441 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 17:29:00.394641 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.394468 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-153.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 17:29:00.394682 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.394646 2576 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 17:29:00.394682 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.394655 2576 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 17:29:00.394737 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.394694 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:29:00.394737 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.394707 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:29:00.395470 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.395460 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 17:29:00.399173 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.399161 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 17:29:00.399286 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.399277 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 17:29:00.399974 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.399964 2576 kubelet.go:493] "Attempting to sync node with API server" Nov 02 17:29:00.400019 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.399978 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 17:29:00.400019 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.399992 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 17:29:00.400019 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.400002 2576 kubelet.go:405] "Adding apiserver pod source" Nov 02 17:29:00.400019 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.400010 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 17:29:00.400374 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.400360 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:29:00.402831 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.402816 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 17:29:00.406827 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.406815 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:29:00.407691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.407677 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 17:29:00.407748 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.407707 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 17:29:00.408343 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.408329 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 17:29:00.408412 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.408346 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 17:29:00.408412 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.408352 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 17:29:00.408412 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.408358 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 17:29:00.408412 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.408364 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 17:29:00.408412 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.408370 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 17:29:00.409077 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.409067 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 17:29:00.409077 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.409077 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 17:29:00.409169 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.409086 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 17:29:00.409169 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.409091 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 17:29:00.409169 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.409109 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 17:29:00.409169 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.409119 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 17:29:00.409890 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.409881 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 17:29:00.409890 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.409890 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 17:29:00.413706 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.413691 2576 server.go:1263] "Started kubelet" Nov 02 17:29:00.413801 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.413769 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 17:29:00.413868 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.413797 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 17:29:00.413912 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.413872 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 17:29:00.414508 ip-10-0-136-153 systemd[1]: Started Kubernetes Kubelet. Nov 02 17:29:00.415356 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.415336 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 17:29:00.421884 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.421838 2576 server.go:317] "Adding debug handlers to kubelet server" Nov 02 17:29:00.422863 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.422609 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 17:29:00.426779 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.426763 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 17:29:00.427361 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.427344 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 17:29:00.427500 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.427475 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-153.ec2.internal\" not found" Nov 02 17:29:00.428096 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.428070 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 17:29:00.428096 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.428079 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 17:29:00.428096 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.428102 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 17:29:00.428340 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.428180 2576 factory.go:55] Registering systemd factory Nov 02 17:29:00.428340 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.428193 2576 factory.go:223] Registration of the systemd container factory successfully Nov 02 17:29:00.428340 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.428211 2576 reconstruct.go:97] "Volume reconstruction finished" Nov 02 17:29:00.428340 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.428218 2576 reconciler.go:29] "Reconciler: start to sync state" Nov 02 17:29:00.428472 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.428420 2576 factory.go:153] Registering CRI-O factory Nov 02 17:29:00.428472 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.428432 2576 factory.go:223] Registration of the crio container factory successfully Nov 02 17:29:00.428535 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.428498 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 17:29:00.428535 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.428514 2576 factory.go:103] Registering Raw factory Nov 02 17:29:00.428535 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.428525 2576 manager.go:1199] Started watching for new ooms in manager Nov 02 17:29:00.430956 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.430467 2576 manager.go:319] Starting recovery of all containers Nov 02 17:29:00.439796 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.439747 2576 manager.go:324] Recovery completed Nov 02 17:29:00.446184 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.446171 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:29:00.448203 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.448185 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasSufficientMemory" Nov 02 17:29:00.448290 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.448220 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 17:29:00.448290 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.448235 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasSufficientPID" Nov 02 17:29:00.448837 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.448819 2576 cpu_manager.go:226] "Starting" policy="none" Nov 02 17:29:00.448837 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.448832 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:29:00.448969 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.448844 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 17:29:00.448969 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.448884 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 17:29:00.449665 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.449647 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasSufficientMemory" Nov 02 17:29:00.449758 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.449680 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 17:29:00.449758 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.449692 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasSufficientPID" Nov 02 17:29:00.451080 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.451069 2576 policy_none.go:50] "Start" Nov 02 17:29:00.451123 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.451083 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 17:29:00.451123 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.451093 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 17:29:00.453006 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.452994 2576 policy_none.go:44] "Start" Nov 02 17:29:00.459405 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.459376 2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-153.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 17:29:00.459467 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.459381 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-153.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 17:29:00.460570 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.459375 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-153.ec2.internal.18e44a918218961c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-153.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-153.ec2.internal,},FirstTimestamp:2026-11-02 17:29:00.413662748 +0000 UTC m=+0.352143937,LastTimestamp:2026-11-02 17:29:00.413662748 +0000 UTC m=+0.352143937,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-153.ec2.internal,}" Nov 02 17:29:00.464233 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.464209 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 17:29:00.478111 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.478040 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-153.ec2.internal.18e44a9184279ecc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-153.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-153.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-153.ec2.internal,},FirstTimestamp:2026-11-02 17:29:00.448202444 +0000 UTC m=+0.386683640,LastTimestamp:2026-11-02 17:29:00.448202444 +0000 UTC m=+0.386683640,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-153.ec2.internal,}" Nov 02 17:29:00.478814 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.478794 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 17:29:00.501391 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.501371 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6n7xq" Nov 02 17:29:00.503322 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.503302 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 17:29:00.503322 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.503274 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-153.ec2.internal.18e44a918427ffd8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-153.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-153.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-153.ec2.internal,},FirstTimestamp:2026-11-02 17:29:00.448227288 +0000 UTC m=+0.386708479,LastTimestamp:2026-11-02 17:29:00.448227288 +0000 UTC m=+0.386708479,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-153.ec2.internal,}" Nov 02 17:29:00.503562 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.503542 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-153.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 17:29:00.509825 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.509808 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6n7xq" Nov 02 17:29:00.518449 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.518384 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-153.ec2.internal.18e44a9184283594 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-153.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-153.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-153.ec2.internal,},FirstTimestamp:2026-11-02 17:29:00.448241044 +0000 UTC m=+0.386722234,LastTimestamp:2026-11-02 17:29:00.448241044 +0000 UTC m=+0.386722234,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-153.ec2.internal,}" Nov 02 17:29:00.540673 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.528071 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-153.ec2.internal\" not found" Nov 02 17:29:00.573630 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.573602 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 17:29:00.574790 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.574772 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 17:29:00.574903 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.574798 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 17:29:00.574903 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.574823 2576 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 17:29:00.574903 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.574878 2576 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 17:29:00.591369 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.591312 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:29:00.628221 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.628203 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-153.ec2.internal\" not found" Nov 02 17:29:00.635382 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.635359 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 17:29:00.637679 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.637667 2576 manager.go:341] "Starting Device Plugin manager" Nov 02 17:29:00.637738 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.637698 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 17:29:00.637738 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.637707 2576 server.go:87] "Starting device plugin registration server" Nov 02 17:29:00.637965 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.637953 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 17:29:00.638028 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.637964 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 17:29:00.638071 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.638055 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 17:29:00.638151 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.638132 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 17:29:00.638151 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.638146 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 17:29:00.638678 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.638655 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 17:29:00.638760 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.638696 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-153.ec2.internal\" not found" Nov 02 17:29:00.675578 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.675557 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal"] Nov 02 17:29:00.675657 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.675608 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:29:00.676569 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.676551 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasSufficientMemory" Nov 02 17:29:00.676647 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.676580 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 17:29:00.676647 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.676591 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasSufficientPID" Nov 02 17:29:00.678801 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.678789 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:29:00.678968 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.678952 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" Nov 02 17:29:00.679043 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.678990 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:29:00.679586 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.679566 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasSufficientMemory" Nov 02 17:29:00.679586 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.679581 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasSufficientMemory" Nov 02 17:29:00.679704 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.679595 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 17:29:00.679704 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.679605 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasSufficientPID" Nov 02 17:29:00.679704 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.679606 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 17:29:00.679704 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.679652 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasSufficientPID" Nov 02 17:29:00.681663 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.681648 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal" Nov 02 17:29:00.681730 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.681671 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:29:00.682385 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.682368 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasSufficientMemory" Nov 02 17:29:00.682481 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.682397 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 17:29:00.682481 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.682412 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasSufficientPID" Nov 02 17:29:00.691325 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.691309 2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-153.ec2.internal\" not found" node="ip-10-0-136-153.ec2.internal" Nov 02 17:29:00.707322 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.707295 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-153.ec2.internal\" not found" node="ip-10-0-136-153.ec2.internal" Nov 02 17:29:00.709959 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.709941 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-153.ec2.internal\" not found" node="ip-10-0-136-153.ec2.internal" Nov 02 17:29:00.729614 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.729594 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6d9161e9f087fe7b62acf0194feffaf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal\" (UID: \"6d9161e9f087fe7b62acf0194feffaf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" Nov 02 17:29:00.729691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.729621 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d9161e9f087fe7b62acf0194feffaf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal\" (UID: \"6d9161e9f087fe7b62acf0194feffaf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" Nov 02 17:29:00.729691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.729637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3f259cc4f9c47e64336e525d3250602-config\") pod \"kube-apiserver-proxy-ip-10-0-136-153.ec2.internal\" (UID: \"e3f259cc4f9c47e64336e525d3250602\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal" Nov 02 17:29:00.738819 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.738801 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:29:00.739609 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.739595 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasSufficientMemory" Nov 02 17:29:00.739668 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.739622 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 17:29:00.739668 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.739632 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasSufficientPID" Nov 02 17:29:00.739668 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.739653 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-153.ec2.internal" Nov 02 17:29:00.762184 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.762168 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-153.ec2.internal" Nov 02 17:29:00.762240 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.762189 2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-153.ec2.internal\": node \"ip-10-0-136-153.ec2.internal\" not found" Nov 02 17:29:00.818440 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.818419 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-153.ec2.internal\" not found" Nov 02 17:29:00.829877 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.829842 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6d9161e9f087fe7b62acf0194feffaf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal\" (UID: \"6d9161e9f087fe7b62acf0194feffaf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" Nov 02 17:29:00.829929 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.829865 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6d9161e9f087fe7b62acf0194feffaf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal\" (UID: \"6d9161e9f087fe7b62acf0194feffaf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" Nov 02 17:29:00.829961 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.829934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d9161e9f087fe7b62acf0194feffaf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal\" (UID: \"6d9161e9f087fe7b62acf0194feffaf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" Nov 02 17:29:00.829992 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.829963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3f259cc4f9c47e64336e525d3250602-config\") pod \"kube-apiserver-proxy-ip-10-0-136-153.ec2.internal\" (UID: \"e3f259cc4f9c47e64336e525d3250602\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal" Nov 02 17:29:00.830024 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.830005 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d9161e9f087fe7b62acf0194feffaf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal\" (UID: \"6d9161e9f087fe7b62acf0194feffaf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" Nov 02 17:29:00.830057 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:00.830039 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3f259cc4f9c47e64336e525d3250602-config\") pod \"kube-apiserver-proxy-ip-10-0-136-153.ec2.internal\" (UID: \"e3f259cc4f9c47e64336e525d3250602\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal" Nov 02 17:29:00.918951 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:00.918897 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-153.ec2.internal\" not found" Nov 02 17:29:01.010284 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:01.010263 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" Nov 02 17:29:01.013410 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:01.013394 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal" Nov 02 17:29:01.019328 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:01.019310 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-153.ec2.internal\" not found" Nov 02 17:29:01.120003 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:01.119967 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-153.ec2.internal\" not found" Nov 02 17:29:01.220488 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:01.220423 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-153.ec2.internal\" not found" Nov 02 17:29:01.321703 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:01.321666 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-153.ec2.internal\" not found" Nov 02 17:29:01.361125 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:01.361083 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 17:29:01.361660 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:01.361243 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 17:29:01.421807 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:01.421772 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-153.ec2.internal\" not found" Nov 02 17:29:01.482135 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:01.482057 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:29:01.515665 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:01.515489 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:24:01 +0000 UTC" deadline="2028-05-10 05:54:21.376617481 +0000 UTC" Nov 02 17:29:01.515665 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:01.515662 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13308h25m19.860959023s" Nov 02 17:29:01.528077 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:01.528056 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" Nov 02 17:29:01.554904 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:01.554876 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:29:01.557000 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:01.556971 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal" Nov 02 17:29:01.577726 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:01.577704 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:29:01.599451 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:01.599422 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:29:01.606724 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:01.606677 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9161e9f087fe7b62acf0194feffaf9.slice/crio-533e0dcf6ba7212004f611f9b2995fc59b1eb2eff0cb758582001a69e7cddbad WatchSource:0}: Error finding container 533e0dcf6ba7212004f611f9b2995fc59b1eb2eff0cb758582001a69e7cddbad: Status 404 returned error can't find the container with id 533e0dcf6ba7212004f611f9b2995fc59b1eb2eff0cb758582001a69e7cddbad Nov 02 17:29:01.607344 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:01.607323 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f259cc4f9c47e64336e525d3250602.slice/crio-f6d4af786dd97bc313a2a112a07af8b16a79f2f3d4ef347307b54d3916c868c6 WatchSource:0}: Error finding container f6d4af786dd97bc313a2a112a07af8b16a79f2f3d4ef347307b54d3916c868c6: Status 404 returned error can't find the container with id f6d4af786dd97bc313a2a112a07af8b16a79f2f3d4ef347307b54d3916c868c6 Nov 02 17:29:01.612633 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:01.612617 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:29:02.078113 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.077997 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:29:02.401374 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.401293 2576 apiserver.go:52] "Watching apiserver" Nov 02 17:29:02.428033 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.428003 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 17:29:02.440938 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.440908 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 17:29:02.441289 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.441267 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-w8vvj","kube-system/konnectivity-agent-vs4r4","openshift-multus/multus-h6dft","openshift-network-operator/iptables-alerter-bmw6p","openshift-dns/node-resolver-s6s58","openshift-network-diagnostics/network-check-target-qjx25","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr","openshift-image-registry/node-ca-jxsgc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal","openshift-multus/multus-additional-cni-plugins-p6d66","kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal","openshift-cluster-node-tuning-operator/tuned-ptdb2","openshift-multus/network-metrics-daemon-g6b2c","openshift-ovn-kubernetes/ovnkube-node-k58q5"] Nov 02 17:29:02.443900 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.443873 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.446081 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.446056 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vs4r4" Nov 02 17:29:02.449179 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.449154 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.452512 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.451777 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bmw6p" Nov 02 17:29:02.455229 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.454480 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.457221 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.456825 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:02.457221 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:02.456938 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qjx25" podUID="97af2a01-5609-4154-a892-5de8ae3e2b9a" Nov 02 17:29:02.459901 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.459619 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:02.459901 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.459730 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 17:29:02.459901 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:02.459741 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6b2c" podUID="4b5c29e0-6bfd-4f43-867a-c460dc227fbc" Nov 02 17:29:02.460522 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.460167 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 17:29:02.460522 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.460225 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 17:29:02.460522 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.460254 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 17:29:02.460522 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.460322 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mwsx7\"" Nov 02 17:29:02.460522 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.460170 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 17:29:02.460522 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.460358 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t2hrp\"" Nov 02 17:29:02.462079 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.462049 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:02.462176 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:02.462148 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8vvj" podUID="b4de14e5-995e-4171-bb61-9f70e777688e" Nov 02 17:29:02.462680 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.462661 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 17:29:02.462867 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.462839 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 17:29:02.463229 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.463208 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 17:29:02.463229 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.463220 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jl8vw\"" Nov 02 17:29:02.463354 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.463224 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 17:29:02.463413 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.463381 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 17:29:02.463413 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.463402 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 17:29:02.469003 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.466636 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:29:02.469003 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.466743 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:02.469003 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.466846 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.469003 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.468780 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4b9fl\"" Nov 02 17:29:02.469003 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.468848 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:02.469003 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.468987 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 17:29:02.469311 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.469080 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 17:29:02.471038 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.470609 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s6s58" Nov 02 17:29:02.471038 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.470905 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 17:29:02.471038 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.470912 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vjnnw\"" Nov 02 17:29:02.473824 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.473562 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 17:29:02.473824 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.473729 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jxsgc" Nov 02 17:29:02.476323 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.475870 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.485846 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.485730 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wt5hc" Nov 02 17:29:02.495691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.495666 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wt5hc" Nov 02 17:29:02.530096 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.530076 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 17:29:02.541334 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.541309 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-sysctl-d\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.541444 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.541349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfjxf\" (UniqueName: \"kubernetes.io/projected/a4b9af78-fe26-40bc-a4df-126ffd0885d2-kube-api-access-mfjxf\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.541444 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.541377 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-log-socket\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.541444 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.541434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-run-ovn-kubernetes\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.541574 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.541467 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.541574 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.541489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvpxv\" (UniqueName: \"kubernetes.io/projected/b3973c00-9abb-42d0-8a2e-6e657af74f2f-kube-api-access-xvpxv\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.541574 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.541519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/daf51d1a-a65f-4fad-8a82-966b8bac7ba4-agent-certs\") pod \"konnectivity-agent-vs4r4\" (UID: \"daf51d1a-a65f-4fad-8a82-966b8bac7ba4\") " pod="kube-system/konnectivity-agent-vs4r4" Nov 02 17:29:02.541703 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.541573 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-var-lib-kubelet\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.541703 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.541603 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-run-multus-certs\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.541703 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.541646 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-kubelet\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.541807 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.541700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-slash\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.541807 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.541748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-cni-netd\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.541807 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.541778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdlkh\" (UniqueName: \"kubernetes.io/projected/4b533156-8ec1-4994-9bf6-75e4c41c3c6a-kube-api-access-zdlkh\") pod \"iptables-alerter-bmw6p\" (UID: \"4b533156-8ec1-4994-9bf6-75e4c41c3c6a\") " pod="openshift-network-operator/iptables-alerter-bmw6p" Nov 02 17:29:02.541928 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.541814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35669b5e-5c97-45ef-8868-4cff3fdfc07a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.541928 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.541843 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9368ef45-b865-4cdf-95a6-5088fc54277e-tmp-dir\") pod \"node-resolver-s6s58\" (UID: \"9368ef45-b865-4cdf-95a6-5088fc54277e\") " pod="openshift-dns/node-resolver-s6s58" Nov 02 17:29:02.541928 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.541883 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z95xd\" (UniqueName: \"kubernetes.io/projected/9368ef45-b865-4cdf-95a6-5088fc54277e-kube-api-access-z95xd\") pod \"node-resolver-s6s58\" (UID: \"9368ef45-b865-4cdf-95a6-5088fc54277e\") " pod="openshift-dns/node-resolver-s6s58" Nov 02 17:29:02.541928 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.541916 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-systemd\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.542093 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.541974 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-cni-dir\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.542093 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542003 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.542093 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542035 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-socket-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.542093 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542076 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35669b5e-5c97-45ef-8868-4cff3fdfc07a-system-cni-dir\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.542221 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-modprobe-d\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.542221 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542128 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-sys\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.542221 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542156 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-system-cni-dir\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.542323 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542259 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-registration-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.542323 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542289 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-run-systemd\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.542323 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542315 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-etc-openvswitch\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.542427 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542343 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35669b5e-5c97-45ef-8868-4cff3fdfc07a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.542427 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-sysconfig\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.542427 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-var-lib-cni-bin\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.542525 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542430 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-device-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.542525 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542471 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3973c00-9abb-42d0-8a2e-6e657af74f2f-ovn-node-metrics-cert\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.542525 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542498 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35669b5e-5c97-45ef-8868-4cff3fdfc07a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.542627 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542525 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-run-k8s-cni-cncf-io\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.542627 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542550 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-node-log\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.542627 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542574 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b3973c00-9abb-42d0-8a2e-6e657af74f2f-ovnkube-config\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.542627 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:02.542774 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542641 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38e5c802-99d4-4234-9e64-dee8e19b055a-tmp\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.542774 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542669 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrzfb\" (UniqueName: \"kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb\") pod \"network-check-target-qjx25\" (UID: \"97af2a01-5609-4154-a892-5de8ae3e2b9a\") " pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:02.542856 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542839 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-run\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.542896 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542876 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:02.542947 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542907 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-run-ovn\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.543005 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542946 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35669b5e-5c97-45ef-8868-4cff3fdfc07a-cnibin\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.543005 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.542996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b44f96b4-5c43-40cc-9c89-63d4ec368e8b-host\") pod \"node-ca-jxsgc\" (UID: \"b44f96b4-5c43-40cc-9c89-63d4ec368e8b\") " pod="openshift-image-registry/node-ca-jxsgc" Nov 02 17:29:02.543162 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.543137 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-hostroot\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.543221 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.543204 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl4xs\" (UniqueName: \"kubernetes.io/projected/c7a1975b-4a3b-426c-977c-8f92d14810ac-kube-api-access-tl4xs\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.543279 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.543244 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-run-netns\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.543317 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.543298 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-run-netns\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.543356 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.543330 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b3973c00-9abb-42d0-8a2e-6e657af74f2f-ovnkube-script-lib\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.543356 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.543349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4de14e5-995e-4171-bb61-9f70e777688e-kubelet-config\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:02.543435 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.543370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-sysctl-conf\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.543435 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.543397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-host\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.543514 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.543464 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jphfq\" (UniqueName: \"kubernetes.io/projected/38e5c802-99d4-4234-9e64-dee8e19b055a-kube-api-access-jphfq\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.543621 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.543604 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-cni-bin\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.543670 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.543648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b3973c00-9abb-42d0-8a2e-6e657af74f2f-env-overrides\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.543732 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.543716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/daf51d1a-a65f-4fad-8a82-966b8bac7ba4-konnectivity-ca\") pod \"konnectivity-agent-vs4r4\" (UID: \"daf51d1a-a65f-4fad-8a82-966b8bac7ba4\") " pod="kube-system/konnectivity-agent-vs4r4" Nov 02 17:29:02.543786 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.543770 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-var-lib-kubelet\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.543826 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.543805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8psf\" (UniqueName: \"kubernetes.io/projected/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-kube-api-access-n8psf\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:02.543877 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.543831 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-cnibin\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.543947 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.543889 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-etc-kubernetes\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.544010 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.543953 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35669b5e-5c97-45ef-8868-4cff3fdfc07a-cni-binary-copy\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.544010 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.543998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-lib-modules\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.544086 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.544047 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm4fn\" (UniqueName: \"kubernetes.io/projected/b44f96b4-5c43-40cc-9c89-63d4ec368e8b-kube-api-access-xm4fn\") pod \"node-ca-jxsgc\" (UID: \"b44f96b4-5c43-40cc-9c89-63d4ec368e8b\") " pod="openshift-image-registry/node-ca-jxsgc" Nov 02 17:29:02.544133 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.544086 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-socket-dir-parent\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.544133 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.544105 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-var-lib-cni-multus\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.544218 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.544192 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-conf-dir\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.544264 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.544230 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-run-openvswitch\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.544302 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.544262 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-daemon-config\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.544302 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.544287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b533156-8ec1-4994-9bf6-75e4c41c3c6a-iptables-alerter-script\") pod \"iptables-alerter-bmw6p\" (UID: \"4b533156-8ec1-4994-9bf6-75e4c41c3c6a\") " pod="openshift-network-operator/iptables-alerter-bmw6p" Nov 02 17:29:02.544397 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.544313 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4de14e5-995e-4171-bb61-9f70e777688e-dbus\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:02.544397 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.544336 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-sys-fs\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.544397 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.544377 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-var-lib-openvswitch\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.544508 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.544441 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b533156-8ec1-4994-9bf6-75e4c41c3c6a-host-slash\") pod \"iptables-alerter-bmw6p\" (UID: \"4b533156-8ec1-4994-9bf6-75e4c41c3c6a\") " pod="openshift-network-operator/iptables-alerter-bmw6p" Nov 02 17:29:02.544508 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.544462 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9368ef45-b865-4cdf-95a6-5088fc54277e-hosts-file\") pod \"node-resolver-s6s58\" (UID: \"9368ef45-b865-4cdf-95a6-5088fc54277e\") " pod="openshift-dns/node-resolver-s6s58" Nov 02 17:29:02.544508 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.544504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-kubernetes\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.544630 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.544521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-tuned\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.544630 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.544536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b44f96b4-5c43-40cc-9c89-63d4ec368e8b-serviceca\") pod \"node-ca-jxsgc\" (UID: \"b44f96b4-5c43-40cc-9c89-63d4ec368e8b\") " pod="openshift-image-registry/node-ca-jxsgc" Nov 02 17:29:02.544630 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.544553 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4b9af78-fe26-40bc-a4df-126ffd0885d2-cni-binary-copy\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.544630 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.544567 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.544630 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.544619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-systemd-units\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.544813 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.544683 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrvfr\" (UniqueName: \"kubernetes.io/projected/35669b5e-5c97-45ef-8868-4cff3fdfc07a-kube-api-access-lrvfr\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.569997 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.569969 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 17:29:02.570755 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.570729 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 17:29:02.570853 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.570841 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 17:29:02.570924 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.570908 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 17:29:02.570986 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.570973 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 17:29:02.571065 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.571052 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 17:29:02.571152 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.571139 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9lngf\"" Nov 02 17:29:02.571197 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.571187 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 17:29:02.571261 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.571249 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 17:29:02.571333 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.571321 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bl7zd\"" Nov 02 17:29:02.571390 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.571380 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 17:29:02.571491 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.571480 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 17:29:02.571539 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.571529 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qbqb7\"" Nov 02 17:29:02.571670 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.571645 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 17:29:02.571797 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.571718 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xdl26\"" Nov 02 17:29:02.582041 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.581985 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" event={"ID":"6d9161e9f087fe7b62acf0194feffaf9","Type":"ContainerStarted","Data":"533e0dcf6ba7212004f611f9b2995fc59b1eb2eff0cb758582001a69e7cddbad"} Nov 02 17:29:02.583417 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.583378 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal" event={"ID":"e3f259cc4f9c47e64336e525d3250602","Type":"ContainerStarted","Data":"f6d4af786dd97bc313a2a112a07af8b16a79f2f3d4ef347307b54d3916c868c6"} Nov 02 17:29:02.645214 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645181 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrzfb\" (UniqueName: \"kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb\") pod \"network-check-target-qjx25\" (UID: \"97af2a01-5609-4154-a892-5de8ae3e2b9a\") " pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:02.645392 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645242 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-run\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.645392 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645290 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:02.645392 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-run-ovn\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.645392 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645341 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35669b5e-5c97-45ef-8868-4cff3fdfc07a-cnibin\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.645392 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b44f96b4-5c43-40cc-9c89-63d4ec368e8b-host\") pod \"node-ca-jxsgc\" (UID: \"b44f96b4-5c43-40cc-9c89-63d4ec368e8b\") " pod="openshift-image-registry/node-ca-jxsgc" Nov 02 17:29:02.645675 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-hostroot\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.645675 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645473 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl4xs\" (UniqueName: \"kubernetes.io/projected/c7a1975b-4a3b-426c-977c-8f92d14810ac-kube-api-access-tl4xs\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.645675 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645491 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-run-netns\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.645675 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-run-netns\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.645675 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b3973c00-9abb-42d0-8a2e-6e657af74f2f-ovnkube-script-lib\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.645675 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4de14e5-995e-4171-bb61-9f70e777688e-kubelet-config\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:02.645675 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-sysctl-conf\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.645675 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645645 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-hostroot\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.646113 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645691 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-host\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.646113 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645712 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jphfq\" (UniqueName: \"kubernetes.io/projected/38e5c802-99d4-4234-9e64-dee8e19b055a-kube-api-access-jphfq\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.646113 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645729 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-cni-bin\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.646113 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645742 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-sysctl-conf\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.646113 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645754 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b3973c00-9abb-42d0-8a2e-6e657af74f2f-env-overrides\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.646113 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/daf51d1a-a65f-4fad-8a82-966b8bac7ba4-konnectivity-ca\") pod \"konnectivity-agent-vs4r4\" (UID: \"daf51d1a-a65f-4fad-8a82-966b8bac7ba4\") " pod="kube-system/konnectivity-agent-vs4r4" Nov 02 17:29:02.646113 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645792 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-var-lib-kubelet\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.646113 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645807 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8psf\" (UniqueName: \"kubernetes.io/projected/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-kube-api-access-n8psf\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:02.646113 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645823 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-cnibin\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.646113 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645838 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-etc-kubernetes\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.646113 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645894 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35669b5e-5c97-45ef-8868-4cff3fdfc07a-cni-binary-copy\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.646113 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-lib-modules\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.646113 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645957 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm4fn\" (UniqueName: \"kubernetes.io/projected/b44f96b4-5c43-40cc-9c89-63d4ec368e8b-kube-api-access-xm4fn\") pod \"node-ca-jxsgc\" (UID: \"b44f96b4-5c43-40cc-9c89-63d4ec368e8b\") " pod="openshift-image-registry/node-ca-jxsgc" Nov 02 17:29:02.646113 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645973 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-socket-dir-parent\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.646113 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.645987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-var-lib-cni-multus\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.646113 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646002 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-conf-dir\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.646113 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646020 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-run-openvswitch\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.646844 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646035 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-daemon-config\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.646844 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b533156-8ec1-4994-9bf6-75e4c41c3c6a-iptables-alerter-script\") pod \"iptables-alerter-bmw6p\" (UID: \"4b533156-8ec1-4994-9bf6-75e4c41c3c6a\") " pod="openshift-network-operator/iptables-alerter-bmw6p" Nov 02 17:29:02.646844 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646069 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4de14e5-995e-4171-bb61-9f70e777688e-dbus\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:02.646844 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646093 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-sys-fs\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.646844 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646113 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-var-lib-openvswitch\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.646844 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646129 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b533156-8ec1-4994-9bf6-75e4c41c3c6a-host-slash\") pod \"iptables-alerter-bmw6p\" (UID: \"4b533156-8ec1-4994-9bf6-75e4c41c3c6a\") " pod="openshift-network-operator/iptables-alerter-bmw6p" Nov 02 17:29:02.646844 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646146 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9368ef45-b865-4cdf-95a6-5088fc54277e-hosts-file\") pod \"node-resolver-s6s58\" (UID: \"9368ef45-b865-4cdf-95a6-5088fc54277e\") " pod="openshift-dns/node-resolver-s6s58" Nov 02 17:29:02.646844 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-kubernetes\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.646844 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646178 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-tuned\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.646844 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b44f96b4-5c43-40cc-9c89-63d4ec368e8b-serviceca\") pod \"node-ca-jxsgc\" (UID: \"b44f96b4-5c43-40cc-9c89-63d4ec368e8b\") " pod="openshift-image-registry/node-ca-jxsgc" Nov 02 17:29:02.646844 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646209 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4b9af78-fe26-40bc-a4df-126ffd0885d2-cni-binary-copy\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.646844 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646234 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.646844 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646251 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-systemd-units\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.646844 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrvfr\" (UniqueName: \"kubernetes.io/projected/35669b5e-5c97-45ef-8868-4cff3fdfc07a-kube-api-access-lrvfr\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.646844 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-sysctl-d\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.646844 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646302 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfjxf\" (UniqueName: \"kubernetes.io/projected/a4b9af78-fe26-40bc-a4df-126ffd0885d2-kube-api-access-mfjxf\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.646844 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646318 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-log-socket\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.647417 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646335 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-run-ovn-kubernetes\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.647417 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.647417 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646371 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvpxv\" (UniqueName: \"kubernetes.io/projected/b3973c00-9abb-42d0-8a2e-6e657af74f2f-kube-api-access-xvpxv\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.647417 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/daf51d1a-a65f-4fad-8a82-966b8bac7ba4-agent-certs\") pod \"konnectivity-agent-vs4r4\" (UID: \"daf51d1a-a65f-4fad-8a82-966b8bac7ba4\") " pod="kube-system/konnectivity-agent-vs4r4" Nov 02 17:29:02.647417 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646404 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-var-lib-kubelet\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.647417 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-run-multus-certs\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.647417 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646435 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-kubelet\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.647417 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646445 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-run-netns\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.647417 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-slash\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.647417 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-cni-netd\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.647417 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdlkh\" (UniqueName: \"kubernetes.io/projected/4b533156-8ec1-4994-9bf6-75e4c41c3c6a-kube-api-access-zdlkh\") pod \"iptables-alerter-bmw6p\" (UID: \"4b533156-8ec1-4994-9bf6-75e4c41c3c6a\") " pod="openshift-network-operator/iptables-alerter-bmw6p" Nov 02 17:29:02.647417 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646521 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35669b5e-5c97-45ef-8868-4cff3fdfc07a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.647417 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9368ef45-b865-4cdf-95a6-5088fc54277e-tmp-dir\") pod \"node-resolver-s6s58\" (UID: \"9368ef45-b865-4cdf-95a6-5088fc54277e\") " pod="openshift-dns/node-resolver-s6s58" Nov 02 17:29:02.647417 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z95xd\" (UniqueName: \"kubernetes.io/projected/9368ef45-b865-4cdf-95a6-5088fc54277e-kube-api-access-z95xd\") pod \"node-resolver-s6s58\" (UID: \"9368ef45-b865-4cdf-95a6-5088fc54277e\") " pod="openshift-dns/node-resolver-s6s58" Nov 02 17:29:02.647417 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646573 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-systemd\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.647417 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646577 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-run\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.647417 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-cni-dir\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.647891 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646613 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.647891 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646630 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-socket-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.647891 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646647 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35669b5e-5c97-45ef-8868-4cff3fdfc07a-system-cni-dir\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.647891 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:02.646653 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:02.647891 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646662 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-modprobe-d\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.647891 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646677 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-sys\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.647891 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:02.646718 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs podName:4b5c29e0-6bfd-4f43-867a-c460dc227fbc nodeName:}" failed. No retries permitted until 2026-11-02 17:29:03.146687832 +0000 UTC m=+3.085169010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs") pod "network-metrics-daemon-g6b2c" (UID: "4b5c29e0-6bfd-4f43-867a-c460dc227fbc") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:02.647891 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646721 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-sys\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.647891 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-system-cni-dir\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.647891 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646751 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-run-netns\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.647891 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646758 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-registration-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.647891 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-run-systemd\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.647891 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-etc-openvswitch\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.647891 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646811 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35669b5e-5c97-45ef-8868-4cff3fdfc07a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.647891 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-sysconfig\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.647891 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646845 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-var-lib-cni-bin\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.647891 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646864 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-device-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.648390 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646882 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3973c00-9abb-42d0-8a2e-6e657af74f2f-ovn-node-metrics-cert\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.648390 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646899 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35669b5e-5c97-45ef-8868-4cff3fdfc07a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.648390 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-run-k8s-cni-cncf-io\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.648390 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646957 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-node-log\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.648390 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646974 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b3973c00-9abb-42d0-8a2e-6e657af74f2f-ovnkube-config\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.648390 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646991 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:02.648390 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.647006 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38e5c802-99d4-4234-9e64-dee8e19b055a-tmp\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.648390 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.647269 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b3973c00-9abb-42d0-8a2e-6e657af74f2f-ovnkube-script-lib\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.648390 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.647330 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 17:29:02.648390 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.647367 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-cni-netd\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.648390 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.646485 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-slash\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.648390 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.647723 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-cni-dir\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.648390 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.647774 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-socket-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.648390 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.647784 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35669b5e-5c97-45ef-8868-4cff3fdfc07a-system-cni-dir\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.648390 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.647838 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.648390 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.647940 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-modprobe-d\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.648390 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.648000 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-systemd\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.648390 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.648089 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35669b5e-5c97-45ef-8868-4cff3fdfc07a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.649051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.648113 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9368ef45-b865-4cdf-95a6-5088fc54277e-tmp-dir\") pod \"node-resolver-s6s58\" (UID: \"9368ef45-b865-4cdf-95a6-5088fc54277e\") " pod="openshift-dns/node-resolver-s6s58" Nov 02 17:29:02.649051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.648241 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b44f96b4-5c43-40cc-9c89-63d4ec368e8b-host\") pod \"node-ca-jxsgc\" (UID: \"b44f96b4-5c43-40cc-9c89-63d4ec368e8b\") " pod="openshift-image-registry/node-ca-jxsgc" Nov 02 17:29:02.649051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.648301 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-run-ovn\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.649051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.648311 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-cni-bin\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.649051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.648342 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35669b5e-5c97-45ef-8868-4cff3fdfc07a-cnibin\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.649051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.648374 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-host\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.649051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.648416 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-var-lib-kubelet\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.649051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.648472 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-system-cni-dir\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.649051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.648557 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.649051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.648595 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-systemd-units\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.649051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.648896 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-device-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.649051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.648976 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-registration-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.649051 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649033 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-sysconfig\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.649691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649080 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-var-lib-cni-bin\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.649691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649128 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b3973c00-9abb-42d0-8a2e-6e657af74f2f-env-overrides\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.649691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649185 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-log-socket\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.649691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649188 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35669b5e-5c97-45ef-8868-4cff3fdfc07a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.649691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649245 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-run-systemd\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.649691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649298 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-etc-openvswitch\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.649691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649301 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-sysctl-d\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.649691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649348 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-run-multus-certs\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.649691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649373 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-kubelet\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.649691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649441 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-run-ovn-kubernetes\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.649691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649509 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-etc-kubernetes\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.649691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649511 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.649691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649510 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4b9af78-fe26-40bc-a4df-126ffd0885d2-cni-binary-copy\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.649691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649546 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35669b5e-5c97-45ef-8868-4cff3fdfc07a-cni-binary-copy\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.649691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649562 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-var-lib-kubelet\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.649691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649572 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-socket-dir-parent\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.649691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649622 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-node-log\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.649691 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649617 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-lib-modules\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.650494 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649633 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-sys-fs\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.650494 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649679 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-conf-dir\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.650494 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649686 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-var-lib-openvswitch\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.650494 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649727 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9368ef45-b865-4cdf-95a6-5088fc54277e-hosts-file\") pod \"node-resolver-s6s58\" (UID: \"9368ef45-b865-4cdf-95a6-5088fc54277e\") " pod="openshift-dns/node-resolver-s6s58" Nov 02 17:29:02.650494 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649735 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-var-lib-cni-multus\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.650494 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649738 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-run-openvswitch\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.650494 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649780 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b533156-8ec1-4994-9bf6-75e4c41c3c6a-host-slash\") pod \"iptables-alerter-bmw6p\" (UID: \"4b533156-8ec1-4994-9bf6-75e4c41c3c6a\") " pod="openshift-network-operator/iptables-alerter-bmw6p" Nov 02 17:29:02.650494 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.647347 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4de14e5-995e-4171-bb61-9f70e777688e-kubelet-config\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:02.650494 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649869 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4de14e5-995e-4171-bb61-9f70e777688e-dbus\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:02.650494 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.649954 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-kubernetes\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.650494 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.650368 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-run-k8s-cni-cncf-io\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.652939 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.651205 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-daemon-config\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.652939 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.651278 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b533156-8ec1-4994-9bf6-75e4c41c3c6a-iptables-alerter-script\") pod \"iptables-alerter-bmw6p\" (UID: \"4b533156-8ec1-4994-9bf6-75e4c41c3c6a\") " pod="openshift-network-operator/iptables-alerter-bmw6p" Nov 02 17:29:02.652939 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.651446 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b3973c00-9abb-42d0-8a2e-6e657af74f2f-ovnkube-config\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.652939 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.652612 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3973c00-9abb-42d0-8a2e-6e657af74f2f-ovn-node-metrics-cert\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.653197 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.653026 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38e5c802-99d4-4234-9e64-dee8e19b055a-tmp\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.653326 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.653302 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/daf51d1a-a65f-4fad-8a82-966b8bac7ba4-konnectivity-ca\") pod \"konnectivity-agent-vs4r4\" (UID: \"daf51d1a-a65f-4fad-8a82-966b8bac7ba4\") " pod="kube-system/konnectivity-agent-vs4r4" Nov 02 17:29:02.653424 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.653382 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-tuned\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.653661 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.653642 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b44f96b4-5c43-40cc-9c89-63d4ec368e8b-serviceca\") pod \"node-ca-jxsgc\" (UID: \"b44f96b4-5c43-40cc-9c89-63d4ec368e8b\") " pod="openshift-image-registry/node-ca-jxsgc" Nov 02 17:29:02.653918 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:02.653808 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:02.653918 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.653837 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35669b5e-5c97-45ef-8868-4cff3fdfc07a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.653918 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:02.653886 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret podName:b4de14e5-995e-4171-bb61-9f70e777688e nodeName:}" failed. No retries permitted until 2026-11-02 17:29:03.153867187 +0000 UTC m=+3.092348382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret") pod "global-pull-secret-syncer-w8vvj" (UID: "b4de14e5-995e-4171-bb61-9f70e777688e") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:02.655091 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.655051 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-cnibin\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.656058 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.655168 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/daf51d1a-a65f-4fad-8a82-966b8bac7ba4-agent-certs\") pod \"konnectivity-agent-vs4r4\" (UID: \"daf51d1a-a65f-4fad-8a82-966b8bac7ba4\") " pod="kube-system/konnectivity-agent-vs4r4" Nov 02 17:29:02.694315 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.694278 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl4xs\" (UniqueName: \"kubernetes.io/projected/c7a1975b-4a3b-426c-977c-8f92d14810ac-kube-api-access-tl4xs\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.697378 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:02.697299 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:29:02.697378 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:02.697335 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:29:02.697378 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:02.697351 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vrzfb for pod openshift-network-diagnostics/network-check-target-qjx25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:02.697593 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:02.697420 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb podName:97af2a01-5609-4154-a892-5de8ae3e2b9a nodeName:}" failed. No retries permitted until 2026-11-02 17:29:03.19739935 +0000 UTC m=+3.135880542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vrzfb" (UniqueName: "kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb") pod "network-check-target-qjx25" (UID: "97af2a01-5609-4154-a892-5de8ae3e2b9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:02.703126 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.703098 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8psf\" (UniqueName: \"kubernetes.io/projected/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-kube-api-access-n8psf\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:02.704141 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.704117 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm4fn\" (UniqueName: \"kubernetes.io/projected/b44f96b4-5c43-40cc-9c89-63d4ec368e8b-kube-api-access-xm4fn\") pod \"node-ca-jxsgc\" (UID: \"b44f96b4-5c43-40cc-9c89-63d4ec368e8b\") " pod="openshift-image-registry/node-ca-jxsgc" Nov 02 17:29:02.711405 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.711377 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrvfr\" (UniqueName: \"kubernetes.io/projected/35669b5e-5c97-45ef-8868-4cff3fdfc07a-kube-api-access-lrvfr\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.712306 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.712283 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z95xd\" (UniqueName: \"kubernetes.io/projected/9368ef45-b865-4cdf-95a6-5088fc54277e-kube-api-access-z95xd\") pod \"node-resolver-s6s58\" (UID: \"9368ef45-b865-4cdf-95a6-5088fc54277e\") " pod="openshift-dns/node-resolver-s6s58" Nov 02 17:29:02.713495 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.713474 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvpxv\" (UniqueName: \"kubernetes.io/projected/b3973c00-9abb-42d0-8a2e-6e657af74f2f-kube-api-access-xvpxv\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.715097 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.715073 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdlkh\" (UniqueName: \"kubernetes.io/projected/4b533156-8ec1-4994-9bf6-75e4c41c3c6a-kube-api-access-zdlkh\") pod \"iptables-alerter-bmw6p\" (UID: \"4b533156-8ec1-4994-9bf6-75e4c41c3c6a\") " pod="openshift-network-operator/iptables-alerter-bmw6p" Nov 02 17:29:02.717775 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.717750 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfjxf\" (UniqueName: \"kubernetes.io/projected/a4b9af78-fe26-40bc-a4df-126ffd0885d2-kube-api-access-mfjxf\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.722707 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.722683 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jphfq\" (UniqueName: \"kubernetes.io/projected/38e5c802-99d4-4234-9e64-dee8e19b055a-kube-api-access-jphfq\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.770377 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.770344 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:02.773020 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.772990 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vs4r4" Nov 02 17:29:02.781570 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:02.781543 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a1975b_4a3b_426c_977c_8f92d14810ac.slice/crio-1f82b9b75a1d021de115fd0758e953cb5e873d00a38ed422f1c672362623e0b0 WatchSource:0}: Error finding container 1f82b9b75a1d021de115fd0758e953cb5e873d00a38ed422f1c672362623e0b0: Status 404 returned error can't find the container with id 1f82b9b75a1d021de115fd0758e953cb5e873d00a38ed422f1c672362623e0b0 Nov 02 17:29:02.785343 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:02.785307 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf51d1a_a65f_4fad_8a82_966b8bac7ba4.slice/crio-d5dd2759fbe8d71f6c45700e5a83443fcb3baac4c517e585e0e7245c0e956460 WatchSource:0}: Error finding container d5dd2759fbe8d71f6c45700e5a83443fcb3baac4c517e585e0e7245c0e956460: Status 404 returned error can't find the container with id d5dd2759fbe8d71f6c45700e5a83443fcb3baac4c517e585e0e7245c0e956460 Nov 02 17:29:02.798406 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.798378 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bmw6p" Nov 02 17:29:02.800631 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.800607 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:02.805069 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.805048 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:02.810220 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:02.810190 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b533156_8ec1_4994_9bf6_75e4c41c3c6a.slice/crio-74924f8e0f80598f0ae2d2226505477ff26bbd3e35ef383ffd97234d57604a1e WatchSource:0}: Error finding container 74924f8e0f80598f0ae2d2226505477ff26bbd3e35ef383ffd97234d57604a1e: Status 404 returned error can't find the container with id 74924f8e0f80598f0ae2d2226505477ff26bbd3e35ef383ffd97234d57604a1e Nov 02 17:29:02.814429 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:02.814403 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e5c802_99d4_4234_9e64_dee8e19b055a.slice/crio-7dcec6338a7bdf13da8301687f0be88a58fb7830e66b2fb19934bcfeb3826777 WatchSource:0}: Error finding container 7dcec6338a7bdf13da8301687f0be88a58fb7830e66b2fb19934bcfeb3826777: Status 404 returned error can't find the container with id 7dcec6338a7bdf13da8301687f0be88a58fb7830e66b2fb19934bcfeb3826777 Nov 02 17:29:02.817850 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:02.817821 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3973c00_9abb_42d0_8a2e_6e657af74f2f.slice/crio-afb720b260f19c89ec5d6fd564551c31801043e60c91fe6cd422f21f3a143d88 WatchSource:0}: Error finding container afb720b260f19c89ec5d6fd564551c31801043e60c91fe6cd422f21f3a143d88: Status 404 returned error can't find the container with id afb720b260f19c89ec5d6fd564551c31801043e60c91fe6cd422f21f3a143d88 Nov 02 17:29:02.820510 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.820487 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s6s58" Nov 02 17:29:02.823794 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.823740 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jxsgc" Nov 02 17:29:02.829076 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.829058 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:02.831238 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:02.831215 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h6dft" Nov 02 17:29:02.841032 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:02.841006 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44f96b4_5c43_40cc_9c89_63d4ec368e8b.slice/crio-5db7566245c6eb0c1454d809ae7c7ffe49dd9a6b0aeb3b2b66136f88fb214087 WatchSource:0}: Error finding container 5db7566245c6eb0c1454d809ae7c7ffe49dd9a6b0aeb3b2b66136f88fb214087: Status 404 returned error can't find the container with id 5db7566245c6eb0c1454d809ae7c7ffe49dd9a6b0aeb3b2b66136f88fb214087 Nov 02 17:29:02.845364 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:02.845339 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35669b5e_5c97_45ef_8868_4cff3fdfc07a.slice/crio-afadebb9c02b6aa05450a3103a4757f749c4382bb8afebb0b9f81979d07c49f7 WatchSource:0}: Error finding container afadebb9c02b6aa05450a3103a4757f749c4382bb8afebb0b9f81979d07c49f7: Status 404 returned error can't find the container with id afadebb9c02b6aa05450a3103a4757f749c4382bb8afebb0b9f81979d07c49f7 Nov 02 17:29:02.845756 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:02.845652 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b9af78_fe26_40bc_a4df_126ffd0885d2.slice/crio-61880416d13b3c700b791ab52cfdc2ebbc6aa31c487736166dc2bbe25b665e32 WatchSource:0}: Error finding container 61880416d13b3c700b791ab52cfdc2ebbc6aa31c487736166dc2bbe25b665e32: Status 404 returned error can't find the container with id 61880416d13b3c700b791ab52cfdc2ebbc6aa31c487736166dc2bbe25b665e32 Nov 02 17:29:03.149912 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:03.149862 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:03.150038 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:03.150019 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:03.150098 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:03.150086 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs podName:4b5c29e0-6bfd-4f43-867a-c460dc227fbc nodeName:}" failed. No retries permitted until 2026-11-02 17:29:04.150066616 +0000 UTC m=+4.088547811 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs") pod "network-metrics-daemon-g6b2c" (UID: "4b5c29e0-6bfd-4f43-867a-c460dc227fbc") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:03.251350 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:03.250963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:03.251350 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:03.251013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrzfb\" (UniqueName: \"kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb\") pod \"network-check-target-qjx25\" (UID: \"97af2a01-5609-4154-a892-5de8ae3e2b9a\") " pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:03.251350 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:03.251183 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:29:03.251350 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:03.251198 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:29:03.251350 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:03.251208 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vrzfb for pod openshift-network-diagnostics/network-check-target-qjx25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:03.251350 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:03.251246 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb podName:97af2a01-5609-4154-a892-5de8ae3e2b9a nodeName:}" failed. No retries permitted until 2026-11-02 17:29:04.251234244 +0000 UTC m=+4.189715437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vrzfb" (UniqueName: "kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb") pod "network-check-target-qjx25" (UID: "97af2a01-5609-4154-a892-5de8ae3e2b9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:03.251350 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:03.251297 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:03.251350 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:03.251328 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret podName:b4de14e5-995e-4171-bb61-9f70e777688e nodeName:}" failed. No retries permitted until 2026-11-02 17:29:04.251319951 +0000 UTC m=+4.189801146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret") pod "global-pull-secret-syncer-w8vvj" (UID: "b4de14e5-995e-4171-bb61-9f70e777688e") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:03.499902 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:03.498916 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:24:03 +0000 UTC" deadline="2028-05-22 02:38:38.989639011 +0000 UTC" Nov 02 17:29:03.499902 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:03.498980 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13593h9m35.490663916s" Nov 02 17:29:03.575057 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:03.575029 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:03.575228 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:03.575166 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qjx25" podUID="97af2a01-5609-4154-a892-5de8ae3e2b9a" Nov 02 17:29:03.591088 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:03.590805 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" event={"ID":"b3973c00-9abb-42d0-8a2e-6e657af74f2f","Type":"ContainerStarted","Data":"afb720b260f19c89ec5d6fd564551c31801043e60c91fe6cd422f21f3a143d88"} Nov 02 17:29:03.601258 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:03.601224 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" event={"ID":"38e5c802-99d4-4234-9e64-dee8e19b055a","Type":"ContainerStarted","Data":"7dcec6338a7bdf13da8301687f0be88a58fb7830e66b2fb19934bcfeb3826777"} Nov 02 17:29:03.610942 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:03.610199 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bmw6p" event={"ID":"4b533156-8ec1-4994-9bf6-75e4c41c3c6a","Type":"ContainerStarted","Data":"74924f8e0f80598f0ae2d2226505477ff26bbd3e35ef383ffd97234d57604a1e"} Nov 02 17:29:03.617720 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:03.617688 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal" event={"ID":"e3f259cc4f9c47e64336e525d3250602","Type":"ContainerStarted","Data":"b5d90f2023d29bdcbffcaee7e9552c5099bcd555864df4d4d2c64d4abaea5c06"} Nov 02 17:29:03.621227 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:03.621198 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h6dft" event={"ID":"a4b9af78-fe26-40bc-a4df-126ffd0885d2","Type":"ContainerStarted","Data":"61880416d13b3c700b791ab52cfdc2ebbc6aa31c487736166dc2bbe25b665e32"} Nov 02 17:29:03.628909 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:03.628883 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6d66" event={"ID":"35669b5e-5c97-45ef-8868-4cff3fdfc07a","Type":"ContainerStarted","Data":"afadebb9c02b6aa05450a3103a4757f749c4382bb8afebb0b9f81979d07c49f7"} Nov 02 17:29:03.639609 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:03.639584 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jxsgc" event={"ID":"b44f96b4-5c43-40cc-9c89-63d4ec368e8b","Type":"ContainerStarted","Data":"5db7566245c6eb0c1454d809ae7c7ffe49dd9a6b0aeb3b2b66136f88fb214087"} Nov 02 17:29:03.643760 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:03.643686 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s6s58" event={"ID":"9368ef45-b865-4cdf-95a6-5088fc54277e","Type":"ContainerStarted","Data":"69bf746fbe676b165b8dc071761c74a42b201c4e3db51d6f2afcd1b14ea86d4e"} Nov 02 17:29:03.653206 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:03.653168 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vs4r4" event={"ID":"daf51d1a-a65f-4fad-8a82-966b8bac7ba4","Type":"ContainerStarted","Data":"d5dd2759fbe8d71f6c45700e5a83443fcb3baac4c517e585e0e7245c0e956460"} Nov 02 17:29:03.657304 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:03.657282 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" event={"ID":"c7a1975b-4a3b-426c-977c-8f92d14810ac","Type":"ContainerStarted","Data":"1f82b9b75a1d021de115fd0758e953cb5e873d00a38ed422f1c672362623e0b0"} Nov 02 17:29:04.156508 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:04.156416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:04.156687 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:04.156629 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:04.156751 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:04.156701 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs podName:4b5c29e0-6bfd-4f43-867a-c460dc227fbc nodeName:}" failed. No retries permitted until 2026-11-02 17:29:06.156679346 +0000 UTC m=+6.095160554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs") pod "network-metrics-daemon-g6b2c" (UID: "4b5c29e0-6bfd-4f43-867a-c460dc227fbc") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:04.258643 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:04.257799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:04.258643 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:04.257852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrzfb\" (UniqueName: \"kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb\") pod \"network-check-target-qjx25\" (UID: \"97af2a01-5609-4154-a892-5de8ae3e2b9a\") " pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:04.258643 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:04.258041 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:29:04.258643 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:04.258065 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:29:04.258643 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:04.258080 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vrzfb for pod openshift-network-diagnostics/network-check-target-qjx25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:04.258643 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:04.258141 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb podName:97af2a01-5609-4154-a892-5de8ae3e2b9a nodeName:}" failed. No retries permitted until 2026-11-02 17:29:06.258121414 +0000 UTC m=+6.196602608 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vrzfb" (UniqueName: "kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb") pod "network-check-target-qjx25" (UID: "97af2a01-5609-4154-a892-5de8ae3e2b9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:04.258643 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:04.258541 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:04.258643 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:04.258597 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret podName:b4de14e5-995e-4171-bb61-9f70e777688e nodeName:}" failed. No retries permitted until 2026-11-02 17:29:06.258581714 +0000 UTC m=+6.197062909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret") pod "global-pull-secret-syncer-w8vvj" (UID: "b4de14e5-995e-4171-bb61-9f70e777688e") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:04.500138 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:04.500046 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:24:03 +0000 UTC" deadline="2028-05-15 02:29:51.049475376 +0000 UTC" Nov 02 17:29:04.500138 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:04.500093 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13425h0m46.549387844s" Nov 02 17:29:04.578219 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:04.578183 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:04.578444 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:04.578342 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6b2c" podUID="4b5c29e0-6bfd-4f43-867a-c460dc227fbc" Nov 02 17:29:04.578774 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:04.578754 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:04.578881 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:04.578858 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8vvj" podUID="b4de14e5-995e-4171-bb61-9f70e777688e" Nov 02 17:29:04.665235 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:04.664226 2576 generic.go:356] "Generic (PLEG): container finished" podID="6d9161e9f087fe7b62acf0194feffaf9" containerID="cffaa3e6baf86786d95ebb88993c6c7957078f225f3f26e540b998ad31595943" exitCode=0 Nov 02 17:29:04.665235 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:04.664991 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" event={"ID":"6d9161e9f087fe7b62acf0194feffaf9","Type":"ContainerDied","Data":"cffaa3e6baf86786d95ebb88993c6c7957078f225f3f26e540b998ad31595943"} Nov 02 17:29:04.707872 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:04.706428 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal" podStartSLOduration=2.706403886 podStartE2EDuration="2.706403886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:03.657337054 +0000 UTC m=+3.595818256" watchObservedRunningTime="2026-11-02 17:29:04.706403886 +0000 UTC m=+4.644885092" Nov 02 17:29:05.575786 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:05.575611 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:05.576155 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:05.575865 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qjx25" podUID="97af2a01-5609-4154-a892-5de8ae3e2b9a" Nov 02 17:29:05.676622 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:05.676588 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" event={"ID":"6d9161e9f087fe7b62acf0194feffaf9","Type":"ContainerStarted","Data":"3d0e91257f8e78ff700016a6613775fb304037d0fbfb1e9add6216e0889104f1"} Nov 02 17:29:06.179955 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:06.176538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:06.179955 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:06.176694 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:06.179955 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:06.176763 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs podName:4b5c29e0-6bfd-4f43-867a-c460dc227fbc nodeName:}" failed. No retries permitted until 2026-11-02 17:29:10.176744624 +0000 UTC m=+10.115225817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs") pod "network-metrics-daemon-g6b2c" (UID: "4b5c29e0-6bfd-4f43-867a-c460dc227fbc") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:06.278081 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:06.277106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:06.278081 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:06.277156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrzfb\" (UniqueName: \"kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb\") pod \"network-check-target-qjx25\" (UID: \"97af2a01-5609-4154-a892-5de8ae3e2b9a\") " pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:06.278081 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:06.277350 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:29:06.278081 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:06.277369 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:29:06.278081 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:06.277383 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vrzfb for pod openshift-network-diagnostics/network-check-target-qjx25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:06.278081 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:06.277442 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb podName:97af2a01-5609-4154-a892-5de8ae3e2b9a nodeName:}" failed. No retries permitted until 2026-11-02 17:29:10.277423125 +0000 UTC m=+10.215904321 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vrzfb" (UniqueName: "kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb") pod "network-check-target-qjx25" (UID: "97af2a01-5609-4154-a892-5de8ae3e2b9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:06.278081 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:06.277880 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:06.278081 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:06.277953 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret podName:b4de14e5-995e-4171-bb61-9f70e777688e nodeName:}" failed. No retries permitted until 2026-11-02 17:29:10.277920487 +0000 UTC m=+10.216401677 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret") pod "global-pull-secret-syncer-w8vvj" (UID: "b4de14e5-995e-4171-bb61-9f70e777688e") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:06.579018 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:06.578010 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:06.579018 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:06.578144 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8vvj" podUID="b4de14e5-995e-4171-bb61-9f70e777688e" Nov 02 17:29:06.579018 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:06.578225 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:06.579018 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:06.578317 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6b2c" podUID="4b5c29e0-6bfd-4f43-867a-c460dc227fbc" Nov 02 17:29:06.692987 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:06.691786 2576 generic.go:356] "Generic (PLEG): container finished" podID="35669b5e-5c97-45ef-8868-4cff3fdfc07a" containerID="21b80520cefbee2633d6adb4be4d11f4252a293fe5a9650107a5cf0af481fe52" exitCode=0 Nov 02 17:29:06.692987 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:06.692763 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6d66" event={"ID":"35669b5e-5c97-45ef-8868-4cff3fdfc07a","Type":"ContainerDied","Data":"21b80520cefbee2633d6adb4be4d11f4252a293fe5a9650107a5cf0af481fe52"} Nov 02 17:29:06.756276 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:06.755647 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" podStartSLOduration=4.755628232 podStartE2EDuration="4.755628232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:05.723497885 +0000 UTC m=+5.661979089" watchObservedRunningTime="2026-11-02 17:29:06.755628232 +0000 UTC m=+6.694109439" Nov 02 17:29:07.575232 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:07.575195 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:07.575409 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:07.575336 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qjx25" podUID="97af2a01-5609-4154-a892-5de8ae3e2b9a" Nov 02 17:29:07.705614 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:07.704656 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vs4r4" event={"ID":"daf51d1a-a65f-4fad-8a82-966b8bac7ba4","Type":"ContainerStarted","Data":"4c6d5f7500af45c5e87299625a5a12492ff4867a2179284eee915e6c4ab2a14e"} Nov 02 17:29:08.579261 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:08.578095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:08.579261 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:08.578241 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6b2c" podUID="4b5c29e0-6bfd-4f43-867a-c460dc227fbc" Nov 02 17:29:08.579261 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:08.578326 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:08.579261 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:08.578409 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8vvj" podUID="b4de14e5-995e-4171-bb61-9f70e777688e" Nov 02 17:29:08.718336 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:08.718264 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jxsgc" event={"ID":"b44f96b4-5c43-40cc-9c89-63d4ec368e8b","Type":"ContainerStarted","Data":"e57a3873b04bd5fe4c16586ea11e5b70db649a680f9f3ec90db4b723ad513a1a"} Nov 02 17:29:08.758957 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:08.758023 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vs4r4" podStartSLOduration=3.78774472 podStartE2EDuration="7.758002583s" totalImagesPullingTime="3.970257863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:07.745213234 +0000 UTC m=+7.683694439" watchObservedRunningTime="2026-11-02 17:29:08.758002583 +0000 UTC m=+8.696483787" Nov 02 17:29:08.758957 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:08.758519 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jxsgc" podStartSLOduration=1.571948726 podStartE2EDuration="6.758510946s" totalImagesPullingTime="5.18656222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:08.757924435 +0000 UTC m=+8.696405640" watchObservedRunningTime="2026-11-02 17:29:08.758510946 +0000 UTC m=+8.696992171" Nov 02 17:29:09.575467 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:09.575388 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:09.575609 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:09.575531 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qjx25" podUID="97af2a01-5609-4154-a892-5de8ae3e2b9a" Nov 02 17:29:09.731219 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:09.731183 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" event={"ID":"c7a1975b-4a3b-426c-977c-8f92d14810ac","Type":"ContainerStarted","Data":"9919ebf8910678920c47b70b9100960fcadcbff544e546c170f3c0aa99c2574b"} Nov 02 17:29:10.214570 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:10.214532 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:10.220412 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:10.218183 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:10.220412 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:10.218274 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs podName:4b5c29e0-6bfd-4f43-867a-c460dc227fbc nodeName:}" failed. No retries permitted until 2026-11-02 17:29:18.218253866 +0000 UTC m=+18.156735059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs") pod "network-metrics-daemon-g6b2c" (UID: "4b5c29e0-6bfd-4f43-867a-c460dc227fbc") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:10.316024 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:10.315976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:10.316199 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:10.316036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrzfb\" (UniqueName: \"kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb\") pod \"network-check-target-qjx25\" (UID: \"97af2a01-5609-4154-a892-5de8ae3e2b9a\") " pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:10.316285 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:10.316228 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:29:10.316285 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:10.316249 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:29:10.316285 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:10.316263 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vrzfb for pod openshift-network-diagnostics/network-check-target-qjx25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:10.316454 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:10.316328 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb podName:97af2a01-5609-4154-a892-5de8ae3e2b9a nodeName:}" failed. No retries permitted until 2026-11-02 17:29:18.316309363 +0000 UTC m=+18.254790547 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vrzfb" (UniqueName: "kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb") pod "network-check-target-qjx25" (UID: "97af2a01-5609-4154-a892-5de8ae3e2b9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:10.316824 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:10.316766 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:10.316954 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:10.316854 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret podName:b4de14e5-995e-4171-bb61-9f70e777688e nodeName:}" failed. No retries permitted until 2026-11-02 17:29:18.31683689 +0000 UTC m=+18.255318091 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret") pod "global-pull-secret-syncer-w8vvj" (UID: "b4de14e5-995e-4171-bb61-9f70e777688e") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:10.416657 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:10.416611 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 17:29:10.579661 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:10.578854 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:10.579661 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:10.579010 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8vvj" podUID="b4de14e5-995e-4171-bb61-9f70e777688e" Nov 02 17:29:10.579661 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:10.579437 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:10.579661 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:10.579549 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6b2c" podUID="4b5c29e0-6bfd-4f43-867a-c460dc227fbc" Nov 02 17:29:10.644043 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:10.643276 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:29:10.416635495Z","UUID":"2cac4829-cddc-4069-bacf-0a27be94f859","Handler":null,"Name":"","Endpoint":""} Nov 02 17:29:10.647484 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:10.647457 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 17:29:10.647621 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:10.647506 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 17:29:10.737850 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:10.737519 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" event={"ID":"c7a1975b-4a3b-426c-977c-8f92d14810ac","Type":"ContainerStarted","Data":"4e7ab3feb6bbbddc28d609c3460470d95a9460af185920c43c744ea4c40e467a"} Nov 02 17:29:11.575192 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:11.574995 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:11.575192 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:11.575145 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qjx25" podUID="97af2a01-5609-4154-a892-5de8ae3e2b9a" Nov 02 17:29:11.743206 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:11.743166 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s6s58" event={"ID":"9368ef45-b865-4cdf-95a6-5088fc54277e","Type":"ContainerStarted","Data":"6ae255e4679b55626e996da9fbcc4e443371122d763f42a53191f051d18960fe"} Nov 02 17:29:11.746240 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:11.746215 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h6dft" event={"ID":"a4b9af78-fe26-40bc-a4df-126ffd0885d2","Type":"ContainerStarted","Data":"e2daa348dead2b4edb74d5f5a1faa4b044f1b57289c0daf27c072d80a2c7da4d"} Nov 02 17:29:11.748483 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:11.748459 2576 generic.go:356] "Generic (PLEG): container finished" podID="35669b5e-5c97-45ef-8868-4cff3fdfc07a" containerID="c29d18cba08ab53e65ed59371193eb27c1ad5ab972a6c38218c88e7131872cb7" exitCode=0 Nov 02 17:29:11.748580 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:11.748499 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6d66" event={"ID":"35669b5e-5c97-45ef-8868-4cff3fdfc07a","Type":"ContainerDied","Data":"c29d18cba08ab53e65ed59371193eb27c1ad5ab972a6c38218c88e7131872cb7"} Nov 02 17:29:11.783690 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:11.783618 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-s6s58" podStartSLOduration=1.763376215 podStartE2EDuration="9.783597956s" totalImagesPullingTime="8.020221741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:11.78228117 +0000 UTC m=+11.720762403" watchObservedRunningTime="2026-11-02 17:29:11.783597956 +0000 UTC m=+11.722079160" Nov 02 17:29:11.883784 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:11.883723 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-h6dft" podStartSLOduration=1.779389122 podStartE2EDuration="9.883699639s" totalImagesPullingTime="8.104310517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:11.883424704 +0000 UTC m=+11.821905910" watchObservedRunningTime="2026-11-02 17:29:11.883699639 +0000 UTC m=+11.822180841" Nov 02 17:29:12.104722 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:12.104682 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vs4r4" Nov 02 17:29:12.105553 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:12.105533 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vs4r4" Nov 02 17:29:12.576454 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:12.575995 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:12.576454 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:12.576030 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:12.576454 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:12.576140 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6b2c" podUID="4b5c29e0-6bfd-4f43-867a-c460dc227fbc" Nov 02 17:29:12.576454 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:12.576271 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8vvj" podUID="b4de14e5-995e-4171-bb61-9f70e777688e" Nov 02 17:29:12.754059 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:12.753982 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" event={"ID":"c7a1975b-4a3b-426c-977c-8f92d14810ac","Type":"ContainerStarted","Data":"564f70f4d38f0ffdc3948b23b28566da68cae3b92d769306706cf9f99415eeea"} Nov 02 17:29:12.757970 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:12.757168 2576 generic.go:356] "Generic (PLEG): container finished" podID="35669b5e-5c97-45ef-8868-4cff3fdfc07a" containerID="aea72000649cfc0d5cdc04e21f53555cc9cc4f95d2af63e8a2ba620f3b91fe5d" exitCode=0 Nov 02 17:29:12.757970 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:12.757277 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6d66" event={"ID":"35669b5e-5c97-45ef-8868-4cff3fdfc07a","Type":"ContainerDied","Data":"aea72000649cfc0d5cdc04e21f53555cc9cc4f95d2af63e8a2ba620f3b91fe5d"} Nov 02 17:29:12.798663 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:12.798608 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" podStartSLOduration=3.401619132 podStartE2EDuration="11.798588899s" totalImagesPullingTime="8.396969767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:12.797640641 +0000 UTC m=+12.736121844" watchObservedRunningTime="2026-11-02 17:29:12.798588899 +0000 UTC m=+12.737070103" Nov 02 17:29:13.225891 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:13.225600 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vs4r4" Nov 02 17:29:13.226274 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:13.226244 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vs4r4" Nov 02 17:29:13.575587 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:13.575503 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:13.575730 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:13.575604 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qjx25" podUID="97af2a01-5609-4154-a892-5de8ae3e2b9a" Nov 02 17:29:13.760815 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:13.760780 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bmw6p" event={"ID":"4b533156-8ec1-4994-9bf6-75e4c41c3c6a","Type":"ContainerStarted","Data":"dba1ef648919b569ff85a09d6cb7919c58f28751b1c8d48e3e7f9f66502a47e7"} Nov 02 17:29:13.825090 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:13.825019 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bmw6p" podStartSLOduration=4.311059312 podStartE2EDuration="12.825001031s" totalImagesPullingTime="8.513941719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:13.824665651 +0000 UTC m=+13.763146874" watchObservedRunningTime="2026-11-02 17:29:13.825001031 +0000 UTC m=+13.763482245" Nov 02 17:29:14.575477 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:14.575443 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:14.575477 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:14.575488 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:14.575691 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:14.575593 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8vvj" podUID="b4de14e5-995e-4171-bb61-9f70e777688e" Nov 02 17:29:14.576019 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:14.575991 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6b2c" podUID="4b5c29e0-6bfd-4f43-867a-c460dc227fbc" Nov 02 17:29:14.764636 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:14.764600 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" event={"ID":"38e5c802-99d4-4234-9e64-dee8e19b055a","Type":"ContainerStarted","Data":"d14e9a687e52a72a7cabbbe12bdff8db0a56bfb0912af5b9789c3a54fff54c22"} Nov 02 17:29:14.768145 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:14.768113 2576 generic.go:356] "Generic (PLEG): container finished" podID="35669b5e-5c97-45ef-8868-4cff3fdfc07a" containerID="cc386ebdd2c93e26c30046b8b86651bc02e26f92708ff389c7b08fde0ee81032" exitCode=0 Nov 02 17:29:14.768285 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:14.768211 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6d66" event={"ID":"35669b5e-5c97-45ef-8868-4cff3fdfc07a","Type":"ContainerDied","Data":"cc386ebdd2c93e26c30046b8b86651bc02e26f92708ff389c7b08fde0ee81032"} Nov 02 17:29:14.822921 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:14.822842 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" podStartSLOduration=2.925926426 podStartE2EDuration="13.822821701s" totalImagesPullingTime="10.896895275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:14.822791772 +0000 UTC m=+14.761272976" watchObservedRunningTime="2026-11-02 17:29:14.822821701 +0000 UTC m=+14.761302903" Nov 02 17:29:15.575786 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:15.575753 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:15.575972 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:15.575873 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qjx25" podUID="97af2a01-5609-4154-a892-5de8ae3e2b9a" Nov 02 17:29:16.575361 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:16.575329 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:16.575951 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:16.575509 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8vvj" podUID="b4de14e5-995e-4171-bb61-9f70e777688e" Nov 02 17:29:16.575951 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:16.575548 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:16.575951 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:16.575676 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6b2c" podUID="4b5c29e0-6bfd-4f43-867a-c460dc227fbc" Nov 02 17:29:16.774669 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:16.774590 2576 generic.go:356] "Generic (PLEG): container finished" podID="35669b5e-5c97-45ef-8868-4cff3fdfc07a" containerID="83fb1ea4639c3dd44d748513affde2f4993ef1eea1f3a91afa1558dda8a18e13" exitCode=0 Nov 02 17:29:16.774669 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:16.774645 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6d66" event={"ID":"35669b5e-5c97-45ef-8868-4cff3fdfc07a","Type":"ContainerDied","Data":"83fb1ea4639c3dd44d748513affde2f4993ef1eea1f3a91afa1558dda8a18e13"} Nov 02 17:29:17.575445 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:17.575246 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:17.575862 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:17.575523 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qjx25" podUID="97af2a01-5609-4154-a892-5de8ae3e2b9a" Nov 02 17:29:17.779596 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:17.779568 2576 generic.go:356] "Generic (PLEG): container finished" podID="35669b5e-5c97-45ef-8868-4cff3fdfc07a" containerID="979dd8973ce0cdc9593a794853beec352f3a2ced6640fb69e611a79c49708ed2" exitCode=0 Nov 02 17:29:17.779782 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:17.779606 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6d66" event={"ID":"35669b5e-5c97-45ef-8868-4cff3fdfc07a","Type":"ContainerDied","Data":"979dd8973ce0cdc9593a794853beec352f3a2ced6640fb69e611a79c49708ed2"} Nov 02 17:29:18.279488 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:18.279460 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:18.279624 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:18.279607 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:18.279670 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:18.279659 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs podName:4b5c29e0-6bfd-4f43-867a-c460dc227fbc nodeName:}" failed. No retries permitted until 2026-11-02 17:29:34.279645787 +0000 UTC m=+34.218126992 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs") pod "network-metrics-daemon-g6b2c" (UID: "4b5c29e0-6bfd-4f43-867a-c460dc227fbc") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:29:18.379923 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:18.379890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:18.380021 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:18.379928 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrzfb\" (UniqueName: \"kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb\") pod \"network-check-target-qjx25\" (UID: \"97af2a01-5609-4154-a892-5de8ae3e2b9a\") " pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:18.380060 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:18.380046 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:18.380060 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:18.380057 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:29:18.380122 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:18.380073 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:29:18.380122 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:18.380085 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vrzfb for pod openshift-network-diagnostics/network-check-target-qjx25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:18.380122 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:18.380107 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret podName:b4de14e5-995e-4171-bb61-9f70e777688e nodeName:}" failed. No retries permitted until 2026-11-02 17:29:34.380088637 +0000 UTC m=+34.318569828 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret") pod "global-pull-secret-syncer-w8vvj" (UID: "b4de14e5-995e-4171-bb61-9f70e777688e") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:29:18.380208 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:18.380123 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb podName:97af2a01-5609-4154-a892-5de8ae3e2b9a nodeName:}" failed. No retries permitted until 2026-11-02 17:29:34.380114669 +0000 UTC m=+34.318595854 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vrzfb" (UniqueName: "kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb") pod "network-check-target-qjx25" (UID: "97af2a01-5609-4154-a892-5de8ae3e2b9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:29:18.575778 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:18.575753 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:18.575778 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:18.575786 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:18.576264 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:18.575893 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6b2c" podUID="4b5c29e0-6bfd-4f43-867a-c460dc227fbc" Nov 02 17:29:18.576264 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:18.575950 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8vvj" podUID="b4de14e5-995e-4171-bb61-9f70e777688e" Nov 02 17:29:18.785240 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:18.785155 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6d66" event={"ID":"35669b5e-5c97-45ef-8868-4cff3fdfc07a","Type":"ContainerStarted","Data":"a484472ccd91c6e74be24b8c7cd137741afcc14c1468d33ac2c7a2ccaa504f5b"} Nov 02 17:29:18.788002 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:18.787978 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" event={"ID":"b3973c00-9abb-42d0-8a2e-6e657af74f2f","Type":"ContainerStarted","Data":"cbae1540141fd07206cb59da448539c7861eba7b56d210b1e03724b3bd92738d"} Nov 02 17:29:18.788097 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:18.788038 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" event={"ID":"b3973c00-9abb-42d0-8a2e-6e657af74f2f","Type":"ContainerStarted","Data":"aa8594eaebfed8e8e6628b432a384ff727f342b99c40bc6e9292000411f06aa7"} Nov 02 17:29:18.788097 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:18.788067 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" event={"ID":"b3973c00-9abb-42d0-8a2e-6e657af74f2f","Type":"ContainerStarted","Data":"8ced43ab9d3cab4c44a1f572869e598a3486a8313a74efb64a2752b2d46c7873"} Nov 02 17:29:18.788097 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:18.788095 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" event={"ID":"b3973c00-9abb-42d0-8a2e-6e657af74f2f","Type":"ContainerStarted","Data":"b3affed1db60bd2c63274e23ce661d3cb6df6f8471e74fc0b638206f6089f7f6"} Nov 02 17:29:18.788192 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:18.788123 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" event={"ID":"b3973c00-9abb-42d0-8a2e-6e657af74f2f","Type":"ContainerStarted","Data":"980f01ffb9b15545746c03efeb5ce7056d46716be592bfea354c1862ae38b557"} Nov 02 17:29:18.788192 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:18.788149 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" event={"ID":"b3973c00-9abb-42d0-8a2e-6e657af74f2f","Type":"ContainerStarted","Data":"6bc27301d58189cc6e140ddf9a114a98572a2123b668d1bcfdbc2f93dad433e4"} Nov 02 17:29:18.830436 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:18.830387 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p6d66" podStartSLOduration=5.715770559 podStartE2EDuration="16.830374634s" totalImagesPullingTime="11.114604075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:18.830226923 +0000 UTC m=+18.768708124" watchObservedRunningTime="2026-11-02 17:29:18.830374634 +0000 UTC m=+18.768855832" Nov 02 17:29:19.576150 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:19.575982 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:19.576596 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:19.576235 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qjx25" podUID="97af2a01-5609-4154-a892-5de8ae3e2b9a" Nov 02 17:29:20.576346 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:20.576303 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:20.576810 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:20.576427 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8vvj" podUID="b4de14e5-995e-4171-bb61-9f70e777688e" Nov 02 17:29:20.576810 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:20.576507 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:20.576810 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:20.576603 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6b2c" podUID="4b5c29e0-6bfd-4f43-867a-c460dc227fbc" Nov 02 17:29:20.796739 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:20.796663 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" event={"ID":"b3973c00-9abb-42d0-8a2e-6e657af74f2f","Type":"ContainerStarted","Data":"5e3e579d570f77d6e7f57181f557ca768cf254c3de6a8f759b9580a378fb43ab"} Nov 02 17:29:21.576030 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:21.575982 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:21.576206 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:21.576097 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qjx25" podUID="97af2a01-5609-4154-a892-5de8ae3e2b9a" Nov 02 17:29:22.575061 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:22.575031 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:22.575586 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:22.575185 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8vvj" podUID="b4de14e5-995e-4171-bb61-9f70e777688e" Nov 02 17:29:22.575586 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:22.575213 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:22.575586 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:22.575339 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6b2c" podUID="4b5c29e0-6bfd-4f43-867a-c460dc227fbc" Nov 02 17:29:22.803529 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:22.803458 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" event={"ID":"b3973c00-9abb-42d0-8a2e-6e657af74f2f","Type":"ContainerStarted","Data":"b0e9993612c144d79fba658f0e4fc1a0ea88c04e34b4cdab1c69a34bb1c7a924"} Nov 02 17:29:22.803712 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:22.803693 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:22.818392 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:22.818369 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:22.896640 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:22.896588 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" podStartSLOduration=6.628643677 podStartE2EDuration="21.896574373s" totalImagesPullingTime="15.267930696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:22.89655211 +0000 UTC m=+22.835033309" watchObservedRunningTime="2026-11-02 17:29:22.896574373 +0000 UTC m=+22.835055563" Nov 02 17:29:23.575452 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:23.575424 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:23.576035 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:23.575516 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qjx25" podUID="97af2a01-5609-4154-a892-5de8ae3e2b9a" Nov 02 17:29:23.805202 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:23.805176 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:29:23.805616 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:23.805597 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:23.820553 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:23.820532 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:24.570567 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:24.570527 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w8vvj"] Nov 02 17:29:24.571047 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:24.571017 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:24.571170 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:24.571144 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8vvj" podUID="b4de14e5-995e-4171-bb61-9f70e777688e" Nov 02 17:29:24.571737 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:24.571713 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qjx25"] Nov 02 17:29:24.571863 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:24.571811 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:24.571943 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:24.571921 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qjx25" podUID="97af2a01-5609-4154-a892-5de8ae3e2b9a" Nov 02 17:29:24.573921 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:24.573891 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g6b2c"] Nov 02 17:29:24.574006 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:24.573977 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:24.574077 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:24.574051 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6b2c" podUID="4b5c29e0-6bfd-4f43-867a-c460dc227fbc" Nov 02 17:29:24.807131 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:24.807106 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:29:25.284567 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:25.284535 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:26.579796 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:26.579630 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:26.580158 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:26.579687 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:26.580158 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:26.579893 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8vvj" podUID="b4de14e5-995e-4171-bb61-9f70e777688e" Nov 02 17:29:26.580158 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:26.579707 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:26.580158 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:26.580003 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6b2c" podUID="4b5c29e0-6bfd-4f43-867a-c460dc227fbc" Nov 02 17:29:26.580158 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:26.580040 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qjx25" podUID="97af2a01-5609-4154-a892-5de8ae3e2b9a" Nov 02 17:29:28.575952 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:28.575919 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:28.575952 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:28.575937 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:28.576353 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:28.576042 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6b2c" podUID="4b5c29e0-6bfd-4f43-867a-c460dc227fbc" Nov 02 17:29:28.576353 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:28.576098 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qjx25" podUID="97af2a01-5609-4154-a892-5de8ae3e2b9a" Nov 02 17:29:28.576353 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:28.576111 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:28.576353 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:28.576188 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8vvj" podUID="b4de14e5-995e-4171-bb61-9f70e777688e" Nov 02 17:29:30.579036 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.579012 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:30.579359 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.579012 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:30.579359 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:30.579110 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6b2c" podUID="4b5c29e0-6bfd-4f43-867a-c460dc227fbc" Nov 02 17:29:30.579359 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.579012 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:30.579359 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:30.579177 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qjx25" podUID="97af2a01-5609-4154-a892-5de8ae3e2b9a" Nov 02 17:29:30.579359 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:30.579245 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8vvj" podUID="b4de14e5-995e-4171-bb61-9f70e777688e" Nov 02 17:29:30.833582 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.833557 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeReady" Nov 02 17:29:30.833734 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.833671 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 17:29:30.905477 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.905449 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hxpgv"] Nov 02 17:29:30.909420 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.909397 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hxpgv" Nov 02 17:29:30.910427 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.910405 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f8ffbdd9-xqf4t"] Nov 02 17:29:30.913641 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.913622 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7jhzf"] Nov 02 17:29:30.913746 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.913730 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:30.916313 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.916295 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7jhzf" Nov 02 17:29:30.921129 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.921109 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nxnkz"] Nov 02 17:29:30.923827 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.923809 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nxnkz" Nov 02 17:29:30.934592 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.934572 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 17:29:30.934797 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.934782 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 17:29:30.956238 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.956216 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 17:29:30.956238 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.956230 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 17:29:30.956368 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.956319 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 17:29:30.956368 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.956328 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mnv5k\"" Nov 02 17:29:30.956425 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.956413 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 17:29:30.962604 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.962586 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w7zsl"] Nov 02 17:29:30.964383 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.963260 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpdh7\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-kube-api-access-rpdh7\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:30.964383 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.963301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slfp9\" (UniqueName: \"kubernetes.io/projected/fbbf8db7-5d42-47b6-b03f-dcd4b68f3211-kube-api-access-slfp9\") pod \"kas-connection-checker-685895c7c9-hxpgv\" (UID: \"fbbf8db7-5d42-47b6-b03f-dcd4b68f3211\") " pod="kube-system/kas-connection-checker-685895c7c9-hxpgv" Nov 02 17:29:30.964383 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.963334 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/27ea270b-6e55-4652-a1bf-1431a21c3d55-ca-trust-extracted\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:30.964383 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.963367 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-image-registry-private-configuration\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:30.964383 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.963390 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-certificates\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:30.964383 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.963411 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-bound-sa-token\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:30.964383 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.963448 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:30.964383 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.963473 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-installation-pull-secrets\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:30.964383 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.963499 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm9gp\" (UniqueName: \"kubernetes.io/projected/e083a23b-76fd-410d-9633-ece3abc240e1-kube-api-access-dm9gp\") pod \"kas-connection-checker-685895c7c9-7jhzf\" (UID: \"e083a23b-76fd-410d-9633-ece3abc240e1\") " pod="kube-system/kas-connection-checker-685895c7c9-7jhzf" Nov 02 17:29:30.964383 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.963521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-trusted-ca\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:30.967467 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.967450 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dmr89"] Nov 02 17:29:30.967593 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.967577 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:30.970142 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.970125 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tszgp"] Nov 02 17:29:30.970257 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.970241 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dmr89" Nov 02 17:29:30.972723 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.972703 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk"] Nov 02 17:29:30.972820 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.972808 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:30.976269 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.975825 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7jhzf"] Nov 02 17:29:30.976269 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.975885 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78749f569b-zzhxm"] Nov 02 17:29:30.977384 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.977365 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:30.979247 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.979226 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-79c77b5cd5-c6svb"] Nov 02 17:29:30.979343 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.979328 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:30.982132 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.982097 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn"] Nov 02 17:29:30.983070 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.983045 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:30.985833 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.985817 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr"] Nov 02 17:29:30.985950 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.985934 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:30.988401 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.988384 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4qfkf"] Nov 02 17:29:30.988529 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.988517 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" Nov 02 17:29:30.991187 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.991164 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7scpf"] Nov 02 17:29:30.992988 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.992967 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4qfkf" Nov 02 17:29:30.994754 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.994737 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-5fvrw"] Nov 02 17:29:30.994996 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.994970 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:30.997580 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.997564 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-xqzqj"] Nov 02 17:29:30.997676 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:30.997663 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5fvrw" Nov 02 17:29:31.000333 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.000316 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn"] Nov 02 17:29:31.000444 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.000426 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:31.003031 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.003013 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f8ffbdd9-xqf4t"] Nov 02 17:29:31.003094 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.003048 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nxnkz"] Nov 02 17:29:31.003094 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.003071 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hxpgv"] Nov 02 17:29:31.003094 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.003091 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w7zsl"] Nov 02 17:29:31.003190 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.003102 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" Nov 02 17:29:31.019134 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.017113 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78749f569b-zzhxm"] Nov 02 17:29:31.021980 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.021954 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr"] Nov 02 17:29:31.022086 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.022010 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7scpf"] Nov 02 17:29:31.023015 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.022982 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4qfkf"] Nov 02 17:29:31.023924 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.023904 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-5fvrw"] Nov 02 17:29:31.024818 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.024799 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn"] Nov 02 17:29:31.025773 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.025756 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-xqzqj"] Nov 02 17:29:31.027714 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.027692 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tszgp"] Nov 02 17:29:31.034139 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.034122 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 17:29:31.034481 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.034465 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 17:29:31.034580 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.034135 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:31.035162 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.035141 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 17:29:31.038603 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.035232 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 17:29:31.038603 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.035258 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 17:29:31.038603 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.035364 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dmr89"] Nov 02 17:29:31.038603 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.035421 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 17:29:31.038603 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.035489 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 17:29:31.038603 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.035519 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 17:29:31.038603 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.035557 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:31.038603 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.035580 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 17:29:31.038603 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.036582 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79c77b5cd5-c6svb"] Nov 02 17:29:31.038603 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.036677 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 17:29:31.038603 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.036685 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 17:29:31.038603 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.036817 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:31.038603 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.036833 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 17:29:31.039244 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.036839 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 17:29:31.039244 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.036891 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:31.039244 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.036907 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 17:29:31.039244 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.036915 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:31.039244 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.036917 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 17:29:31.039244 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.036980 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 17:29:31.039244 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.037004 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 17:29:31.039244 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.037129 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 17:29:31.039244 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.037290 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 17:29:31.039244 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.037350 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 17:29:31.039244 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.037406 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 17:29:31.039686 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.037489 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 17:29:31.039686 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.037524 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 17:29:31.045220 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.045194 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 17:29:31.047509 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.047486 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 17:29:31.047774 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.047747 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 17:29:31.047912 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.047822 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 17:29:31.048792 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.048769 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 17:29:31.048913 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.048820 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 17:29:31.049994 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.049971 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ncmj6\"" Nov 02 17:29:31.050075 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.050012 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 17:29:31.050141 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.050087 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-qz666\"" Nov 02 17:29:31.050203 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.050142 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 17:29:31.050203 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.050166 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-q8x9s\"" Nov 02 17:29:31.050203 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.050175 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 17:29:31.050203 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.050192 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 17:29:31.050386 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.050230 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 17:29:31.050386 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.050279 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 17:29:31.050386 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.050290 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-npw6n\"" Nov 02 17:29:31.050386 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.050318 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 17:29:31.050386 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.050345 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 17:29:31.050386 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.050357 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-c5qgh\"" Nov 02 17:29:31.050649 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.050391 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-chwwq\"" Nov 02 17:29:31.050649 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.050446 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5nstj\"" Nov 02 17:29:31.050649 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.050459 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9d6qd\"" Nov 02 17:29:31.050649 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.050505 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9bnd5\"" Nov 02 17:29:31.050649 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.050527 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wbstr\"" Nov 02 17:29:31.051405 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.051384 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 17:29:31.051487 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.051468 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rj9zp\"" Nov 02 17:29:31.051541 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.051480 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk"] Nov 02 17:29:31.059267 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.059247 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn"] Nov 02 17:29:31.063827 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.063803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5f9e8821-40a8-4012-b66a-813e80d856f2-config\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:31.063939 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.063894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc4a6db8-98e0-4387-8e64-1020978970c3-serving-cert\") pod \"service-ca-operator-544d749c8c-zbfmn\" (UID: \"bc4a6db8-98e0-4387-8e64-1020978970c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" Nov 02 17:29:31.063939 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.063924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a453427b-5025-4653-a930-ce2cf2f41cce-tmp-dir\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:31.064049 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.063989 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spg94\" (UniqueName: \"kubernetes.io/projected/5f9e8821-40a8-4012-b66a-813e80d856f2-kube-api-access-spg94\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:31.064049 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064021 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-certificates\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:31.064122 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064064 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-bound-sa-token\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:31.064174 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064128 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:31.064174 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:31.064263 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064193 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:31.064263 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064237 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw824\" (UniqueName: \"kubernetes.io/projected/bc4a6db8-98e0-4387-8e64-1020978970c3-kube-api-access-cw824\") pod \"service-ca-operator-544d749c8c-zbfmn\" (UID: \"bc4a6db8-98e0-4387-8e64-1020978970c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" Nov 02 17:29:31.064355 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064321 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tdtz\" (UniqueName: \"kubernetes.io/projected/f20b07b4-cd27-4410-abb2-c0bd50c01269-kube-api-access-9tdtz\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:31.064410 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064367 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f9e8821-40a8-4012-b66a-813e80d856f2-trusted-ca\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:31.064460 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064435 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:31.064508 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064467 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-installation-pull-secrets\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:31.064508 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7scpf\" (UID: \"72445bf5-a7a3-49dc-9b97-5589ae901032\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:31.064608 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064508 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7484d61b-5158-4729-b835-c0b372d8105b-ca-trust-extracted\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.064608 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064524 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-installation-pull-secrets\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.064608 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b4dt\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-kube-api-access-2b4dt\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.064608 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064569 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/704c917e-c41f-47c3-9a1e-02730361fc8a-cert\") pod \"ingress-canary-4qfkf\" (UID: \"704c917e-c41f-47c3-9a1e-02730361fc8a\") " pod="openshift-ingress-canary/ingress-canary-4qfkf" Nov 02 17:29:31.064608 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064583 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-image-registry-private-configuration\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.064608 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064598 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-registry-certificates\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.064949 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064614 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-bound-sa-token\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.064949 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpdh7\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-kube-api-access-rpdh7\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:31.064949 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064657 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd9gl\" (UniqueName: \"kubernetes.io/projected/704c917e-c41f-47c3-9a1e-02730361fc8a-kube-api-access-jd9gl\") pod \"ingress-canary-4qfkf\" (UID: \"704c917e-c41f-47c3-9a1e-02730361fc8a\") " pod="openshift-ingress-canary/ingress-canary-4qfkf" Nov 02 17:29:31.064949 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064672 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-stats-auth\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:31.064949 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064687 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm8pr\" (UniqueName: \"kubernetes.io/projected/3d83bf78-e7b7-48bb-98be-8196e92fd320-kube-api-access-mm8pr\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:31.064949 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064687 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-certificates\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:31.064949 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kqjw\" (UniqueName: \"kubernetes.io/projected/a074a9ac-624c-4828-9785-c8f2aecb5827-kube-api-access-5kqjw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr\" (UID: \"a074a9ac-624c-4828-9785-c8f2aecb5827\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" Nov 02 17:29:31.064949 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfp9\" (UniqueName: \"kubernetes.io/projected/fbbf8db7-5d42-47b6-b03f-dcd4b68f3211-kube-api-access-slfp9\") pod \"kas-connection-checker-685895c7c9-hxpgv\" (UID: \"fbbf8db7-5d42-47b6-b03f-dcd4b68f3211\") " pod="kube-system/kas-connection-checker-685895c7c9-hxpgv" Nov 02 17:29:31.064949 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064777 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:31.064949 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064794 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jf4x\" (UniqueName: \"kubernetes.io/projected/42f9e736-411e-46a2-89d0-2a3870e2f95a-kube-api-access-8jf4x\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:31.064949 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqrgs\" (UniqueName: \"kubernetes.io/projected/46d2456f-62b2-4aa5-b172-1c28ab7615ac-kube-api-access-wqrgs\") pod \"kas-connection-checker-685895c7c9-nxnkz\" (UID: \"46d2456f-62b2-4aa5-b172-1c28ab7615ac\") " pod="kube-system/kas-connection-checker-685895c7c9-nxnkz" Nov 02 17:29:31.064949 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064830 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt4dk\" (UniqueName: \"kubernetes.io/projected/7e4dc8ce-2802-4864-aed5-affeefe41d6b-kube-api-access-pt4dk\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:31.064949 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064864 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a453427b-5025-4653-a930-ce2cf2f41cce-config-volume\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:31.064949 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a074a9ac-624c-4828-9785-c8f2aecb5827-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr\" (UID: \"a074a9ac-624c-4828-9785-c8f2aecb5827\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" Nov 02 17:29:31.064949 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064925 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a074a9ac-624c-4828-9785-c8f2aecb5827-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr\" (UID: \"a074a9ac-624c-4828-9785-c8f2aecb5827\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" Nov 02 17:29:31.065533 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064951 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-image-registry-private-configuration\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:31.065533 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.064984 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-trusted-ca\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.065533 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.065018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g4g4\" (UniqueName: \"kubernetes.io/projected/f7a7133f-8372-49a3-8194-15ed2159aa67-kube-api-access-4g4g4\") pod \"volume-data-source-validator-58c9f5f46d-dmr89\" (UID: \"f7a7133f-8372-49a3-8194-15ed2159aa67\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dmr89" Nov 02 17:29:31.065533 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.065272 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 17:29:31.065533 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.065412 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8ffbdd9-xqf4t: secret "image-registry-tls" not found Nov 02 17:29:31.065810 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.065785 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f9e8821-40a8-4012-b66a-813e80d856f2-serving-cert\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:31.065900 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.065842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/42f9e736-411e-46a2-89d0-2a3870e2f95a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:31.066068 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.066049 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:31.066121 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.066094 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm9gp\" (UniqueName: \"kubernetes.io/projected/e083a23b-76fd-410d-9633-ece3abc240e1-kube-api-access-dm9gp\") pod \"kas-connection-checker-685895c7c9-7jhzf\" (UID: \"e083a23b-76fd-410d-9633-ece3abc240e1\") " pod="kube-system/kas-connection-checker-685895c7c9-7jhzf" Nov 02 17:29:31.066588 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.066133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-trusted-ca\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:31.067211 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.067191 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls podName:27ea270b-6e55-4652-a1bf-1431a21c3d55 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:31.567061454 +0000 UTC m=+31.505542643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls") pod "image-registry-6f8ffbdd9-xqf4t" (UID: "27ea270b-6e55-4652-a1bf-1431a21c3d55") : secret "image-registry-tls" not found Nov 02 17:29:31.067952 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.067924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/72445bf5-a7a3-49dc-9b97-5589ae901032-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7scpf\" (UID: \"72445bf5-a7a3-49dc-9b97-5589ae901032\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:31.068167 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.068083 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njp5v\" (UniqueName: \"kubernetes.io/projected/0eacac81-35d0-4ab1-bea4-25aaefa74c8b-kube-api-access-njp5v\") pod \"network-check-source-6f78d59f99-5fvrw\" (UID: \"0eacac81-35d0-4ab1-bea4-25aaefa74c8b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5fvrw" Nov 02 17:29:31.068227 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.068194 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7e4dc8ce-2802-4864-aed5-affeefe41d6b-snapshots\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:31.068282 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.068232 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-trusted-ca\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:31.068614 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.068415 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e4dc8ce-2802-4864-aed5-affeefe41d6b-tmp\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:31.068614 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.068546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4m8s\" (UniqueName: \"kubernetes.io/projected/a453427b-5025-4653-a930-ce2cf2f41cce-kube-api-access-m4m8s\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:31.068747 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.068647 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bc4a6db8-98e0-4387-8e64-1020978970c3-config\") pod \"service-ca-operator-544d749c8c-zbfmn\" (UID: \"bc4a6db8-98e0-4387-8e64-1020978970c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" Nov 02 17:29:31.068869 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.068826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e4dc8ce-2802-4864-aed5-affeefe41d6b-service-ca-bundle\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:31.069137 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.069115 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7e4dc8ce-2802-4864-aed5-affeefe41d6b-serving-cert\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:31.069202 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.069185 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:31.069354 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.069310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/27ea270b-6e55-4652-a1bf-1431a21c3d55-ca-trust-extracted\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:31.069477 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.069460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.069775 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.069750 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/27ea270b-6e55-4652-a1bf-1431a21c3d55-ca-trust-extracted\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:31.069878 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.069817 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-default-certificate\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:31.069932 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.069887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e4dc8ce-2802-4864-aed5-affeefe41d6b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:31.069932 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.069906 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-image-registry-private-configuration\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:31.070902 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.070879 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-installation-pull-secrets\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:31.087683 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.087615 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slfp9\" (UniqueName: \"kubernetes.io/projected/fbbf8db7-5d42-47b6-b03f-dcd4b68f3211-kube-api-access-slfp9\") pod \"kas-connection-checker-685895c7c9-hxpgv\" (UID: \"fbbf8db7-5d42-47b6-b03f-dcd4b68f3211\") " pod="kube-system/kas-connection-checker-685895c7c9-hxpgv" Nov 02 17:29:31.088179 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.088160 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-bound-sa-token\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:31.095092 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.095069 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpdh7\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-kube-api-access-rpdh7\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:31.097987 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.097968 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm9gp\" (UniqueName: \"kubernetes.io/projected/e083a23b-76fd-410d-9633-ece3abc240e1-kube-api-access-dm9gp\") pod \"kas-connection-checker-685895c7c9-7jhzf\" (UID: \"e083a23b-76fd-410d-9633-ece3abc240e1\") " pod="kube-system/kas-connection-checker-685895c7c9-7jhzf" Nov 02 17:29:31.170580 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.170550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7e4dc8ce-2802-4864-aed5-affeefe41d6b-serving-cert\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:31.170580 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.170579 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:31.170801 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.170596 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.170801 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.170617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-default-certificate\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:31.170801 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.170641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e4dc8ce-2802-4864-aed5-affeefe41d6b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:31.170801 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.170666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5f9e8821-40a8-4012-b66a-813e80d856f2-config\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:31.170801 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.170690 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc4a6db8-98e0-4387-8e64-1020978970c3-serving-cert\") pod \"service-ca-operator-544d749c8c-zbfmn\" (UID: \"bc4a6db8-98e0-4387-8e64-1020978970c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" Nov 02 17:29:31.170801 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.170724 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle podName:3d83bf78-e7b7-48bb-98be-8196e92fd320 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:31.670702151 +0000 UTC m=+31.609183345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle") pod "router-default-79c77b5cd5-c6svb" (UID: "3d83bf78-e7b7-48bb-98be-8196e92fd320") : configmap references non-existent config key: service-ca.crt Nov 02 17:29:31.170801 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.170736 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 17:29:31.170801 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.170757 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78749f569b-zzhxm: secret "image-registry-tls" not found Nov 02 17:29:31.170801 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.170757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a453427b-5025-4653-a930-ce2cf2f41cce-tmp-dir\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:31.170801 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.170790 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spg94\" (UniqueName: \"kubernetes.io/projected/5f9e8821-40a8-4012-b66a-813e80d856f2-kube-api-access-spg94\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:31.170801 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.170809 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls podName:7484d61b-5158-4729-b835-c0b372d8105b nodeName:}" failed. No retries permitted until 2026-11-02 17:29:31.670792387 +0000 UTC m=+31.609273579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls") pod "image-registry-78749f569b-zzhxm" (UID: "7484d61b-5158-4729-b835-c0b372d8105b") : secret "image-registry-tls" not found Nov 02 17:29:31.171362 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.170876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:31.171362 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.170908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:31.171362 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.170937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:31.171362 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.170967 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw824\" (UniqueName: \"kubernetes.io/projected/bc4a6db8-98e0-4387-8e64-1020978970c3-kube-api-access-cw824\") pod \"service-ca-operator-544d749c8c-zbfmn\" (UID: \"bc4a6db8-98e0-4387-8e64-1020978970c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" Nov 02 17:29:31.171362 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tdtz\" (UniqueName: \"kubernetes.io/projected/f20b07b4-cd27-4410-abb2-c0bd50c01269-kube-api-access-9tdtz\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:31.171362 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171027 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f9e8821-40a8-4012-b66a-813e80d856f2-trusted-ca\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:31.171362 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7scpf\" (UID: \"72445bf5-a7a3-49dc-9b97-5589ae901032\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:31.171362 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7484d61b-5158-4729-b835-c0b372d8105b-ca-trust-extracted\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.171362 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-installation-pull-secrets\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.171362 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.171153 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 17:29:31.171362 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b4dt\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-kube-api-access-2b4dt\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.171362 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/704c917e-c41f-47c3-9a1e-02730361fc8a-cert\") pod \"ingress-canary-4qfkf\" (UID: \"704c917e-c41f-47c3-9a1e-02730361fc8a\") " pod="openshift-ingress-canary/ingress-canary-4qfkf" Nov 02 17:29:31.171362 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171221 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a453427b-5025-4653-a930-ce2cf2f41cce-tmp-dir\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:31.171362 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-image-registry-private-configuration\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.171362 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171256 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-registry-certificates\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.171362 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-bound-sa-token\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.173152 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd9gl\" (UniqueName: \"kubernetes.io/projected/704c917e-c41f-47c3-9a1e-02730361fc8a-kube-api-access-jd9gl\") pod \"ingress-canary-4qfkf\" (UID: \"704c917e-c41f-47c3-9a1e-02730361fc8a\") " pod="openshift-ingress-canary/ingress-canary-4qfkf" Nov 02 17:29:31.173152 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171330 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-stats-auth\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:31.173152 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.171355 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls podName:42f9e736-411e-46a2-89d0-2a3870e2f95a nodeName:}" failed. No retries permitted until 2026-11-02 17:29:31.671337282 +0000 UTC m=+31.609818471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pthgk" (UID: "42f9e736-411e-46a2-89d0-2a3870e2f95a") : secret "cluster-monitoring-operator-tls" not found Nov 02 17:29:31.173152 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171390 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm8pr\" (UniqueName: \"kubernetes.io/projected/3d83bf78-e7b7-48bb-98be-8196e92fd320-kube-api-access-mm8pr\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:31.173152 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kqjw\" (UniqueName: \"kubernetes.io/projected/a074a9ac-624c-4828-9785-c8f2aecb5827-kube-api-access-5kqjw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr\" (UID: \"a074a9ac-624c-4828-9785-c8f2aecb5827\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" Nov 02 17:29:31.173152 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171465 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:31.173152 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171492 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jf4x\" (UniqueName: \"kubernetes.io/projected/42f9e736-411e-46a2-89d0-2a3870e2f95a-kube-api-access-8jf4x\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:31.173152 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqrgs\" (UniqueName: \"kubernetes.io/projected/46d2456f-62b2-4aa5-b172-1c28ab7615ac-kube-api-access-wqrgs\") pod \"kas-connection-checker-685895c7c9-nxnkz\" (UID: \"46d2456f-62b2-4aa5-b172-1c28ab7615ac\") " pod="kube-system/kas-connection-checker-685895c7c9-nxnkz" Nov 02 17:29:31.173152 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt4dk\" (UniqueName: \"kubernetes.io/projected/7e4dc8ce-2802-4864-aed5-affeefe41d6b-kube-api-access-pt4dk\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:31.173152 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171588 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e4dc8ce-2802-4864-aed5-affeefe41d6b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:31.173152 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a453427b-5025-4653-a930-ce2cf2f41cce-config-volume\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:31.173152 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171630 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a074a9ac-624c-4828-9785-c8f2aecb5827-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr\" (UID: \"a074a9ac-624c-4828-9785-c8f2aecb5827\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" Nov 02 17:29:31.173152 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171656 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a074a9ac-624c-4828-9785-c8f2aecb5827-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr\" (UID: \"a074a9ac-624c-4828-9785-c8f2aecb5827\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" Nov 02 17:29:31.173152 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171721 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-trusted-ca\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.174055 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4g4\" (UniqueName: \"kubernetes.io/projected/f7a7133f-8372-49a3-8194-15ed2159aa67-kube-api-access-4g4g4\") pod \"volume-data-source-validator-58c9f5f46d-dmr89\" (UID: \"f7a7133f-8372-49a3-8194-15ed2159aa67\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dmr89" Nov 02 17:29:31.174055 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f9e8821-40a8-4012-b66a-813e80d856f2-serving-cert\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:31.174055 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171798 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/42f9e736-411e-46a2-89d0-2a3870e2f95a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:31.174055 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:31.174055 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171873 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/72445bf5-a7a3-49dc-9b97-5589ae901032-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7scpf\" (UID: \"72445bf5-a7a3-49dc-9b97-5589ae901032\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:31.174055 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njp5v\" (UniqueName: \"kubernetes.io/projected/0eacac81-35d0-4ab1-bea4-25aaefa74c8b-kube-api-access-njp5v\") pod \"network-check-source-6f78d59f99-5fvrw\" (UID: \"0eacac81-35d0-4ab1-bea4-25aaefa74c8b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5fvrw" Nov 02 17:29:31.174055 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171905 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:31.174055 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171927 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7e4dc8ce-2802-4864-aed5-affeefe41d6b-snapshots\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:31.174055 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e4dc8ce-2802-4864-aed5-affeefe41d6b-tmp\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:31.174055 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171994 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4m8s\" (UniqueName: \"kubernetes.io/projected/a453427b-5025-4653-a930-ce2cf2f41cce-kube-api-access-m4m8s\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:31.174055 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.172022 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bc4a6db8-98e0-4387-8e64-1020978970c3-config\") pod \"service-ca-operator-544d749c8c-zbfmn\" (UID: \"bc4a6db8-98e0-4387-8e64-1020978970c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" Nov 02 17:29:31.174055 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.172047 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e4dc8ce-2802-4864-aed5-affeefe41d6b-service-ca-bundle\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:31.174055 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.172157 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7484d61b-5158-4729-b835-c0b372d8105b-ca-trust-extracted\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.174055 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.172250 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 17:29:31.174055 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.172293 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs podName:3d83bf78-e7b7-48bb-98be-8196e92fd320 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:31.672279233 +0000 UTC m=+31.610760410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs") pod "router-default-79c77b5cd5-c6svb" (UID: "3d83bf78-e7b7-48bb-98be-8196e92fd320") : secret "router-metrics-certs-default" not found Nov 02 17:29:31.174055 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.172568 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-registry-certificates\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.174904 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.172628 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e4dc8ce-2802-4864-aed5-affeefe41d6b-service-ca-bundle\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:31.174904 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.172726 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 17:29:31.174904 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.172773 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert podName:72445bf5-a7a3-49dc-9b97-5589ae901032 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:31.672758574 +0000 UTC m=+31.611239756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7scpf" (UID: "72445bf5-a7a3-49dc-9b97-5589ae901032") : secret "networking-console-plugin-cert" not found Nov 02 17:29:31.174904 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.173555 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f9e8821-40a8-4012-b66a-813e80d856f2-trusted-ca\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:31.174904 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.173675 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc4a6db8-98e0-4387-8e64-1020978970c3-serving-cert\") pod \"service-ca-operator-544d749c8c-zbfmn\" (UID: \"bc4a6db8-98e0-4387-8e64-1020978970c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" Nov 02 17:29:31.174904 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.173925 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 17:29:31.174904 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.173980 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls podName:a453427b-5025-4653-a930-ce2cf2f41cce nodeName:}" failed. No retries permitted until 2026-11-02 17:29:31.673967062 +0000 UTC m=+31.612448242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls") pod "dns-default-w7zsl" (UID: "a453427b-5025-4653-a930-ce2cf2f41cce") : secret "dns-default-metrics-tls" not found Nov 02 17:29:31.174904 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.174078 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 17:29:31.174904 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.174125 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls podName:f20b07b4-cd27-4410-abb2-c0bd50c01269 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:31.674110711 +0000 UTC m=+31.612591902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dxkqn" (UID: "f20b07b4-cd27-4410-abb2-c0bd50c01269") : secret "samples-operator-tls" not found Nov 02 17:29:31.174904 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.174215 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-installation-pull-secrets\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.174904 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.171463 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5f9e8821-40a8-4012-b66a-813e80d856f2-config\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:31.174904 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.174355 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/704c917e-c41f-47c3-9a1e-02730361fc8a-cert\") pod \"ingress-canary-4qfkf\" (UID: \"704c917e-c41f-47c3-9a1e-02730361fc8a\") " pod="openshift-ingress-canary/ingress-canary-4qfkf" Nov 02 17:29:31.174904 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.174457 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/42f9e736-411e-46a2-89d0-2a3870e2f95a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:31.174904 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.174759 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7e4dc8ce-2802-4864-aed5-affeefe41d6b-snapshots\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:31.174904 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.174825 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a074a9ac-624c-4828-9785-c8f2aecb5827-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr\" (UID: \"a074a9ac-624c-4828-9785-c8f2aecb5827\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" Nov 02 17:29:31.175619 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.175125 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/72445bf5-a7a3-49dc-9b97-5589ae901032-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7scpf\" (UID: \"72445bf5-a7a3-49dc-9b97-5589ae901032\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:31.175619 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.175146 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7e4dc8ce-2802-4864-aed5-affeefe41d6b-serving-cert\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:31.175619 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.175366 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bc4a6db8-98e0-4387-8e64-1020978970c3-config\") pod \"service-ca-operator-544d749c8c-zbfmn\" (UID: \"bc4a6db8-98e0-4387-8e64-1020978970c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" Nov 02 17:29:31.175619 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.175512 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-stats-auth\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:31.175915 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.175802 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-image-registry-private-configuration\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.175915 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.175868 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e4dc8ce-2802-4864-aed5-affeefe41d6b-tmp\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:31.176085 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.176062 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a453427b-5025-4653-a930-ce2cf2f41cce-config-volume\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:31.176242 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.176222 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-trusted-ca\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.176778 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.176763 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-default-certificate\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:31.177223 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.177207 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f9e8821-40a8-4012-b66a-813e80d856f2-serving-cert\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:31.177677 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.177663 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a074a9ac-624c-4828-9785-c8f2aecb5827-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr\" (UID: \"a074a9ac-624c-4828-9785-c8f2aecb5827\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" Nov 02 17:29:31.202958 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.202937 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd9gl\" (UniqueName: \"kubernetes.io/projected/704c917e-c41f-47c3-9a1e-02730361fc8a-kube-api-access-jd9gl\") pod \"ingress-canary-4qfkf\" (UID: \"704c917e-c41f-47c3-9a1e-02730361fc8a\") " pod="openshift-ingress-canary/ingress-canary-4qfkf" Nov 02 17:29:31.203877 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.203840 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqrgs\" (UniqueName: \"kubernetes.io/projected/46d2456f-62b2-4aa5-b172-1c28ab7615ac-kube-api-access-wqrgs\") pod \"kas-connection-checker-685895c7c9-nxnkz\" (UID: \"46d2456f-62b2-4aa5-b172-1c28ab7615ac\") " pod="kube-system/kas-connection-checker-685895c7c9-nxnkz" Nov 02 17:29:31.203975 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.203960 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-bound-sa-token\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.204630 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.204614 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm8pr\" (UniqueName: \"kubernetes.io/projected/3d83bf78-e7b7-48bb-98be-8196e92fd320-kube-api-access-mm8pr\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:31.208714 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.208411 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njp5v\" (UniqueName: \"kubernetes.io/projected/0eacac81-35d0-4ab1-bea4-25aaefa74c8b-kube-api-access-njp5v\") pod \"network-check-source-6f78d59f99-5fvrw\" (UID: \"0eacac81-35d0-4ab1-bea4-25aaefa74c8b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5fvrw" Nov 02 17:29:31.209174 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.209150 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4m8s\" (UniqueName: \"kubernetes.io/projected/a453427b-5025-4653-a930-ce2cf2f41cce-kube-api-access-m4m8s\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:31.209404 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.209383 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tdtz\" (UniqueName: \"kubernetes.io/projected/f20b07b4-cd27-4410-abb2-c0bd50c01269-kube-api-access-9tdtz\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:31.210279 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.210256 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw824\" (UniqueName: \"kubernetes.io/projected/bc4a6db8-98e0-4387-8e64-1020978970c3-kube-api-access-cw824\") pod \"service-ca-operator-544d749c8c-zbfmn\" (UID: \"bc4a6db8-98e0-4387-8e64-1020978970c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" Nov 02 17:29:31.210540 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.210524 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b4dt\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-kube-api-access-2b4dt\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.210701 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.210685 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt4dk\" (UniqueName: \"kubernetes.io/projected/7e4dc8ce-2802-4864-aed5-affeefe41d6b-kube-api-access-pt4dk\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:31.216444 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.216424 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jf4x\" (UniqueName: \"kubernetes.io/projected/42f9e736-411e-46a2-89d0-2a3870e2f95a-kube-api-access-8jf4x\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:31.216768 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.216752 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spg94\" (UniqueName: \"kubernetes.io/projected/5f9e8821-40a8-4012-b66a-813e80d856f2-kube-api-access-spg94\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:31.219063 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.219044 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4g4\" (UniqueName: \"kubernetes.io/projected/f7a7133f-8372-49a3-8194-15ed2159aa67-kube-api-access-4g4g4\") pod \"volume-data-source-validator-58c9f5f46d-dmr89\" (UID: \"f7a7133f-8372-49a3-8194-15ed2159aa67\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dmr89" Nov 02 17:29:31.221978 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.221965 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hxpgv" Nov 02 17:29:31.225750 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.225732 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kqjw\" (UniqueName: \"kubernetes.io/projected/a074a9ac-624c-4828-9785-c8f2aecb5827-kube-api-access-5kqjw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr\" (UID: \"a074a9ac-624c-4828-9785-c8f2aecb5827\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" Nov 02 17:29:31.233827 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.233811 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7jhzf" Nov 02 17:29:31.238529 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.238514 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nxnkz" Nov 02 17:29:31.292820 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.292729 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dmr89" Nov 02 17:29:31.299826 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.299786 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:31.312768 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.312713 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5fvrw" Nov 02 17:29:31.347367 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.347339 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" Nov 02 17:29:31.356033 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.355944 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4qfkf" Nov 02 17:29:31.373768 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.373132 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:31.381174 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.381006 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" Nov 02 17:29:31.544184 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.544156 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7jhzf"] Nov 02 17:29:31.553971 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:31.553934 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode083a23b_76fd_410d_9633_ece3abc240e1.slice/crio-c35565f0de21fb64553bae63d86fa9635c3e543017dd84bf2ba17486215cda7a WatchSource:0}: Error finding container c35565f0de21fb64553bae63d86fa9635c3e543017dd84bf2ba17486215cda7a: Status 404 returned error can't find the container with id c35565f0de21fb64553bae63d86fa9635c3e543017dd84bf2ba17486215cda7a Nov 02 17:29:31.560921 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.560893 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hxpgv"] Nov 02 17:29:31.573136 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.572716 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nxnkz"] Nov 02 17:29:31.576260 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.575504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:31.576260 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.575633 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 17:29:31.576260 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.575648 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8ffbdd9-xqf4t: secret "image-registry-tls" not found Nov 02 17:29:31.576260 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.575687 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls podName:27ea270b-6e55-4652-a1bf-1431a21c3d55 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:32.575671275 +0000 UTC m=+32.514152495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls") pod "image-registry-6f8ffbdd9-xqf4t" (UID: "27ea270b-6e55-4652-a1bf-1431a21c3d55") : secret "image-registry-tls" not found Nov 02 17:29:31.639363 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.639288 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dmr89"] Nov 02 17:29:31.652964 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:31.652935 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a7133f_8372_49a3_8194_15ed2159aa67.slice/crio-4072df9c777140fe13d8e46d3c3c46ce4d4842f2cdb8f0e9ad3db97af69c50a3 WatchSource:0}: Error finding container 4072df9c777140fe13d8e46d3c3c46ce4d4842f2cdb8f0e9ad3db97af69c50a3: Status 404 returned error can't find the container with id 4072df9c777140fe13d8e46d3c3c46ce4d4842f2cdb8f0e9ad3db97af69c50a3 Nov 02 17:29:31.661036 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.660923 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tszgp"] Nov 02 17:29:31.666965 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:31.665986 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4dc8ce_2802_4864_aed5_affeefe41d6b.slice/crio-9c38e5e87ea0feaac1a2bf835454b3c0968e9442f8fba1d8225f9b3565583717 WatchSource:0}: Error finding container 9c38e5e87ea0feaac1a2bf835454b3c0968e9442f8fba1d8225f9b3565583717: Status 404 returned error can't find the container with id 9c38e5e87ea0feaac1a2bf835454b3c0968e9442f8fba1d8225f9b3565583717 Nov 02 17:29:31.672565 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.672227 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-5fvrw"] Nov 02 17:29:31.677156 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.676144 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:31.677156 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.676190 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:31.677156 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.676263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7scpf\" (UID: \"72445bf5-a7a3-49dc-9b97-5589ae901032\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:31.677156 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.676329 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 17:29:31.677156 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.676354 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:31.677156 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.676410 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls podName:42f9e736-411e-46a2-89d0-2a3870e2f95a nodeName:}" failed. No retries permitted until 2026-11-02 17:29:32.676390422 +0000 UTC m=+32.614871618 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pthgk" (UID: "42f9e736-411e-46a2-89d0-2a3870e2f95a") : secret "cluster-monitoring-operator-tls" not found Nov 02 17:29:31.677156 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.676465 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 17:29:31.677156 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.676513 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls podName:f20b07b4-cd27-4410-abb2-c0bd50c01269 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:32.676498459 +0000 UTC m=+32.614979642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dxkqn" (UID: "f20b07b4-cd27-4410-abb2-c0bd50c01269") : secret "samples-operator-tls" not found Nov 02 17:29:31.677156 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.676515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:31.677156 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.676580 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 17:29:31.677156 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.676617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:31.677156 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.676639 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs podName:3d83bf78-e7b7-48bb-98be-8196e92fd320 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:32.676604653 +0000 UTC m=+32.615085850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs") pod "router-default-79c77b5cd5-c6svb" (UID: "3d83bf78-e7b7-48bb-98be-8196e92fd320") : secret "router-metrics-certs-default" not found Nov 02 17:29:31.677156 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.676725 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 17:29:31.677156 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.676760 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle podName:3d83bf78-e7b7-48bb-98be-8196e92fd320 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:32.676727105 +0000 UTC m=+32.615208282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle") pod "router-default-79c77b5cd5-c6svb" (UID: "3d83bf78-e7b7-48bb-98be-8196e92fd320") : configmap references non-existent config key: service-ca.crt Nov 02 17:29:31.677156 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.676780 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls podName:a453427b-5025-4653-a930-ce2cf2f41cce nodeName:}" failed. No retries permitted until 2026-11-02 17:29:32.67677037 +0000 UTC m=+32.615251550 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls") pod "dns-default-w7zsl" (UID: "a453427b-5025-4653-a930-ce2cf2f41cce") : secret "dns-default-metrics-tls" not found Nov 02 17:29:31.678271 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.676799 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 17:29:31.678271 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.676808 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:31.678271 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.677030 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 17:29:31.678271 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.677046 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78749f569b-zzhxm: secret "image-registry-tls" not found Nov 02 17:29:31.678271 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.677084 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls podName:7484d61b-5158-4729-b835-c0b372d8105b nodeName:}" failed. No retries permitted until 2026-11-02 17:29:32.677071189 +0000 UTC m=+32.615552376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls") pod "image-registry-78749f569b-zzhxm" (UID: "7484d61b-5158-4729-b835-c0b372d8105b") : secret "image-registry-tls" not found Nov 02 17:29:31.678271 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:31.677343 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert podName:72445bf5-a7a3-49dc-9b97-5589ae901032 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:32.677328021 +0000 UTC m=+32.615809202 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7scpf" (UID: "72445bf5-a7a3-49dc-9b97-5589ae901032") : secret "networking-console-plugin-cert" not found Nov 02 17:29:31.678960 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:31.678933 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eacac81_35d0_4ab1_bea4_25aaefa74c8b.slice/crio-de2b882663fbc211a07a1f8327255abab18faeefc69d5113bffae1f9a25149c8 WatchSource:0}: Error finding container de2b882663fbc211a07a1f8327255abab18faeefc69d5113bffae1f9a25149c8: Status 404 returned error can't find the container with id de2b882663fbc211a07a1f8327255abab18faeefc69d5113bffae1f9a25149c8 Nov 02 17:29:31.719492 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.719463 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr"] Nov 02 17:29:31.723912 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:31.723886 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda074a9ac_624c_4828_9785_c8f2aecb5827.slice/crio-1bdf615d401fa21d9ac512784085320be2cb538e86cc41992d168f339019b37e WatchSource:0}: Error finding container 1bdf615d401fa21d9ac512784085320be2cb538e86cc41992d168f339019b37e: Status 404 returned error can't find the container with id 1bdf615d401fa21d9ac512784085320be2cb538e86cc41992d168f339019b37e Nov 02 17:29:31.728974 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.728948 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4qfkf"] Nov 02 17:29:31.733348 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:31.733299 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704c917e_c41f_47c3_9a1e_02730361fc8a.slice/crio-f5e336b15af15f9570964afe8da5675163e1868f984595421ce2c9f2a07cb7e2 WatchSource:0}: Error finding container f5e336b15af15f9570964afe8da5675163e1868f984595421ce2c9f2a07cb7e2: Status 404 returned error can't find the container with id f5e336b15af15f9570964afe8da5675163e1868f984595421ce2c9f2a07cb7e2 Nov 02 17:29:31.741751 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.741318 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-xqzqj"] Nov 02 17:29:31.742273 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.742253 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn"] Nov 02 17:29:31.744293 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:31.744266 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9e8821_40a8_4012_b66a_813e80d856f2.slice/crio-19e5c90135094488bf4817a4198f7e14939531ab670190f348241b436254c4f1 WatchSource:0}: Error finding container 19e5c90135094488bf4817a4198f7e14939531ab670190f348241b436254c4f1: Status 404 returned error can't find the container with id 19e5c90135094488bf4817a4198f7e14939531ab670190f348241b436254c4f1 Nov 02 17:29:31.747272 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:31.747249 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc4a6db8_98e0_4387_8e64_1020978970c3.slice/crio-cd8bb8092f4aed3a7e68285f1a30d1bcaae6ae0c5568d26f8f9bb3a5643cdd85 WatchSource:0}: Error finding container cd8bb8092f4aed3a7e68285f1a30d1bcaae6ae0c5568d26f8f9bb3a5643cdd85: Status 404 returned error can't find the container with id cd8bb8092f4aed3a7e68285f1a30d1bcaae6ae0c5568d26f8f9bb3a5643cdd85 Nov 02 17:29:31.822482 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.822452 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4qfkf" event={"ID":"704c917e-c41f-47c3-9a1e-02730361fc8a","Type":"ContainerStarted","Data":"f5e336b15af15f9570964afe8da5675163e1868f984595421ce2c9f2a07cb7e2"} Nov 02 17:29:31.823554 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.823528 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" event={"ID":"bc4a6db8-98e0-4387-8e64-1020978970c3","Type":"ContainerStarted","Data":"cd8bb8092f4aed3a7e68285f1a30d1bcaae6ae0c5568d26f8f9bb3a5643cdd85"} Nov 02 17:29:31.824532 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.824513 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5fvrw" event={"ID":"0eacac81-35d0-4ab1-bea4-25aaefa74c8b","Type":"ContainerStarted","Data":"de2b882663fbc211a07a1f8327255abab18faeefc69d5113bffae1f9a25149c8"} Nov 02 17:29:31.825537 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.825511 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" event={"ID":"a074a9ac-624c-4828-9785-c8f2aecb5827","Type":"ContainerStarted","Data":"1bdf615d401fa21d9ac512784085320be2cb538e86cc41992d168f339019b37e"} Nov 02 17:29:31.826907 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.826885 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hxpgv" event={"ID":"fbbf8db7-5d42-47b6-b03f-dcd4b68f3211","Type":"ContainerStarted","Data":"b44c50de67e40f4a8c10afd0ba9bd5597ce609d1726a289ed5cc20c78992624c"} Nov 02 17:29:31.826998 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.826920 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hxpgv" event={"ID":"fbbf8db7-5d42-47b6-b03f-dcd4b68f3211","Type":"ContainerStarted","Data":"dfff4b53674ec62275631a314edb707b7e7f3be4efccafa74bd017d5f347212e"} Nov 02 17:29:31.828055 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.828034 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dmr89" event={"ID":"f7a7133f-8372-49a3-8194-15ed2159aa67","Type":"ContainerStarted","Data":"4072df9c777140fe13d8e46d3c3c46ce4d4842f2cdb8f0e9ad3db97af69c50a3"} Nov 02 17:29:31.829341 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.829324 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7jhzf" event={"ID":"e083a23b-76fd-410d-9633-ece3abc240e1","Type":"ContainerStarted","Data":"d3913bb934e7a49fc27d203dda5665033f36bb3a9b0ff67ef4b6b730ee9b690f"} Nov 02 17:29:31.829423 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.829351 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7jhzf" event={"ID":"e083a23b-76fd-410d-9633-ece3abc240e1","Type":"ContainerStarted","Data":"c35565f0de21fb64553bae63d86fa9635c3e543017dd84bf2ba17486215cda7a"} Nov 02 17:29:31.830416 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.830388 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tszgp" event={"ID":"7e4dc8ce-2802-4864-aed5-affeefe41d6b","Type":"ContainerStarted","Data":"9c38e5e87ea0feaac1a2bf835454b3c0968e9442f8fba1d8225f9b3565583717"} Nov 02 17:29:31.831725 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.831705 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nxnkz" event={"ID":"46d2456f-62b2-4aa5-b172-1c28ab7615ac","Type":"ContainerStarted","Data":"492e25d8acdbaf114fe6d66e6a34d6b56661a2bdc0014a4199f7fb2aacabd6c2"} Nov 02 17:29:31.831793 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.831737 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nxnkz" event={"ID":"46d2456f-62b2-4aa5-b172-1c28ab7615ac","Type":"ContainerStarted","Data":"732370e3f644cd5f750ef139deb4731284af5b0051284347014685a23ed3b579"} Nov 02 17:29:31.832789 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.832770 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" event={"ID":"5f9e8821-40a8-4012-b66a-813e80d856f2","Type":"ContainerStarted","Data":"19e5c90135094488bf4817a4198f7e14939531ab670190f348241b436254c4f1"} Nov 02 17:29:31.864505 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.864438 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-hxpgv" podStartSLOduration=441.864427556 podStartE2EDuration="7m21.864427556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:31.863519912 +0000 UTC m=+31.802001111" watchObservedRunningTime="2026-11-02 17:29:31.864427556 +0000 UTC m=+31.802908791" Nov 02 17:29:31.903140 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:31.903102 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7jhzf" podStartSLOduration=441.903090715 podStartE2EDuration="7m21.903090715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:31.901501094 +0000 UTC m=+31.839982292" watchObservedRunningTime="2026-11-02 17:29:31.903090715 +0000 UTC m=+31.841571914" Nov 02 17:29:32.575810 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:32.575768 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:32.577355 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:32.577335 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:32.577905 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:32.577890 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:32.591720 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:32.591673 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:32.592011 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:32.591994 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 17:29:32.592084 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:32.592015 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8ffbdd9-xqf4t: secret "image-registry-tls" not found Nov 02 17:29:32.592084 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:32.592070 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls podName:27ea270b-6e55-4652-a1bf-1431a21c3d55 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:34.592051684 +0000 UTC m=+34.530532875 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls") pod "image-registry-6f8ffbdd9-xqf4t" (UID: "27ea270b-6e55-4652-a1bf-1431a21c3d55") : secret "image-registry-tls" not found Nov 02 17:29:32.597437 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:32.596917 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 17:29:32.597437 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:32.597063 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zt2z5\"" Nov 02 17:29:32.597437 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:32.597160 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ppwdb\"" Nov 02 17:29:32.597437 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:32.597285 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 17:29:32.597437 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:32.597395 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nn2p2\"" Nov 02 17:29:32.694075 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:32.692698 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:32.694075 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:32.692766 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:32.694075 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:32.692796 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:32.694075 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:32.692843 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7scpf\" (UID: \"72445bf5-a7a3-49dc-9b97-5589ae901032\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:32.694075 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:32.692926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:32.694075 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:32.692982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:32.694075 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:32.693032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:32.694075 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:32.693193 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 17:29:32.694075 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:32.693256 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs podName:3d83bf78-e7b7-48bb-98be-8196e92fd320 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:34.693237157 +0000 UTC m=+34.631718356 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs") pod "router-default-79c77b5cd5-c6svb" (UID: "3d83bf78-e7b7-48bb-98be-8196e92fd320") : secret "router-metrics-certs-default" not found Nov 02 17:29:32.694075 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:32.693646 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 17:29:32.694075 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:32.693660 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78749f569b-zzhxm: secret "image-registry-tls" not found Nov 02 17:29:32.694075 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:32.693703 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls podName:7484d61b-5158-4729-b835-c0b372d8105b nodeName:}" failed. No retries permitted until 2026-11-02 17:29:34.693689317 +0000 UTC m=+34.632170500 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls") pod "image-registry-78749f569b-zzhxm" (UID: "7484d61b-5158-4729-b835-c0b372d8105b") : secret "image-registry-tls" not found Nov 02 17:29:32.694075 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:32.693757 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 17:29:32.694075 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:32.693788 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls podName:42f9e736-411e-46a2-89d0-2a3870e2f95a nodeName:}" failed. No retries permitted until 2026-11-02 17:29:34.693778236 +0000 UTC m=+34.632259431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pthgk" (UID: "42f9e736-411e-46a2-89d0-2a3870e2f95a") : secret "cluster-monitoring-operator-tls" not found Nov 02 17:29:32.694075 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:32.693863 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 17:29:32.694075 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:32.693895 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls podName:f20b07b4-cd27-4410-abb2-c0bd50c01269 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:34.693885012 +0000 UTC m=+34.632366205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dxkqn" (UID: "f20b07b4-cd27-4410-abb2-c0bd50c01269") : secret "samples-operator-tls" not found Nov 02 17:29:32.695212 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:32.693945 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 17:29:32.695212 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:32.693974 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert podName:72445bf5-a7a3-49dc-9b97-5589ae901032 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:34.693964069 +0000 UTC m=+34.632445261 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7scpf" (UID: "72445bf5-a7a3-49dc-9b97-5589ae901032") : secret "networking-console-plugin-cert" not found Nov 02 17:29:32.695212 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:32.694024 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 17:29:32.695212 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:32.694051 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls podName:a453427b-5025-4653-a930-ce2cf2f41cce nodeName:}" failed. No retries permitted until 2026-11-02 17:29:34.694042597 +0000 UTC m=+34.632523776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls") pod "dns-default-w7zsl" (UID: "a453427b-5025-4653-a930-ce2cf2f41cce") : secret "dns-default-metrics-tls" not found Nov 02 17:29:32.695533 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:32.695506 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle podName:3d83bf78-e7b7-48bb-98be-8196e92fd320 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:34.69549352 +0000 UTC m=+34.633974697 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle") pod "router-default-79c77b5cd5-c6svb" (UID: "3d83bf78-e7b7-48bb-98be-8196e92fd320") : configmap references non-existent config key: service-ca.crt Nov 02 17:29:34.310543 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:34.310508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:34.310992 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:34.310722 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 17:29:34.310992 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:34.310783 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs podName:4b5c29e0-6bfd-4f43-867a-c460dc227fbc nodeName:}" failed. No retries permitted until 2026-11-02 17:30:06.31076543 +0000 UTC m=+66.249246697 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs") pod "network-metrics-daemon-g6b2c" (UID: "4b5c29e0-6bfd-4f43-867a-c460dc227fbc") : secret "metrics-daemon-secret" not found Nov 02 17:29:34.412525 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:34.411481 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:34.412525 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:34.411526 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrzfb\" (UniqueName: \"kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb\") pod \"network-check-target-qjx25\" (UID: \"97af2a01-5609-4154-a892-5de8ae3e2b9a\") " pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:34.424143 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:34.424044 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrzfb\" (UniqueName: \"kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb\") pod \"network-check-target-qjx25\" (UID: \"97af2a01-5609-4154-a892-5de8ae3e2b9a\") " pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:34.426489 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:34.426459 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:34.463965 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:34.463757 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:34.614250 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:34.614032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:34.615091 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:34.614535 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 17:29:34.615091 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:34.614562 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8ffbdd9-xqf4t: secret "image-registry-tls" not found Nov 02 17:29:34.615091 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:34.614623 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls podName:27ea270b-6e55-4652-a1bf-1431a21c3d55 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:38.614603895 +0000 UTC m=+38.553085089 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls") pod "image-registry-6f8ffbdd9-xqf4t" (UID: "27ea270b-6e55-4652-a1bf-1431a21c3d55") : secret "image-registry-tls" not found Nov 02 17:29:34.713220 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:34.713130 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:34.715823 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:34.715791 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:34.716912 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:34.715836 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:34.716912 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:34.715897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7scpf\" (UID: \"72445bf5-a7a3-49dc-9b97-5589ae901032\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:34.716912 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:34.715941 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:34.716912 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:34.715984 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 17:29:34.716912 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:34.716058 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls podName:42f9e736-411e-46a2-89d0-2a3870e2f95a nodeName:}" failed. No retries permitted until 2026-11-02 17:29:38.716036946 +0000 UTC m=+38.654518150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pthgk" (UID: "42f9e736-411e-46a2-89d0-2a3870e2f95a") : secret "cluster-monitoring-operator-tls" not found Nov 02 17:29:34.716912 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:34.715993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:34.716912 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:34.716081 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 17:29:34.716912 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:34.716126 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls podName:a453427b-5025-4653-a930-ce2cf2f41cce nodeName:}" failed. No retries permitted until 2026-11-02 17:29:38.716111848 +0000 UTC m=+38.654593030 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls") pod "dns-default-w7zsl" (UID: "a453427b-5025-4653-a930-ce2cf2f41cce") : secret "dns-default-metrics-tls" not found Nov 02 17:29:34.716912 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:34.716158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:34.716912 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:34.716190 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:34.716912 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:34.716292 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 17:29:34.716912 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:34.716304 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78749f569b-zzhxm: secret "image-registry-tls" not found Nov 02 17:29:34.716912 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:34.716337 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls podName:7484d61b-5158-4729-b835-c0b372d8105b nodeName:}" failed. No retries permitted until 2026-11-02 17:29:38.716326072 +0000 UTC m=+38.654807252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls") pod "image-registry-78749f569b-zzhxm" (UID: "7484d61b-5158-4729-b835-c0b372d8105b") : secret "image-registry-tls" not found Nov 02 17:29:34.716912 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:34.716407 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 17:29:34.716912 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:34.716437 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs podName:3d83bf78-e7b7-48bb-98be-8196e92fd320 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:38.716426018 +0000 UTC m=+38.654907199 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs") pod "router-default-79c77b5cd5-c6svb" (UID: "3d83bf78-e7b7-48bb-98be-8196e92fd320") : secret "router-metrics-certs-default" not found Nov 02 17:29:34.716912 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:34.716525 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 17:29:34.717697 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:34.716593 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert podName:72445bf5-a7a3-49dc-9b97-5589ae901032 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:38.716565944 +0000 UTC m=+38.655047127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7scpf" (UID: "72445bf5-a7a3-49dc-9b97-5589ae901032") : secret "networking-console-plugin-cert" not found Nov 02 17:29:34.717697 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:34.716622 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle podName:3d83bf78-e7b7-48bb-98be-8196e92fd320 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:38.716606989 +0000 UTC m=+38.655088171 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle") pod "router-default-79c77b5cd5-c6svb" (UID: "3d83bf78-e7b7-48bb-98be-8196e92fd320") : configmap references non-existent config key: service-ca.crt Nov 02 17:29:34.717697 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:34.716671 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 17:29:34.717697 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:34.716709 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls podName:f20b07b4-cd27-4410-abb2-c0bd50c01269 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:38.716696528 +0000 UTC m=+38.655177736 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dxkqn" (UID: "f20b07b4-cd27-4410-abb2-c0bd50c01269") : secret "samples-operator-tls" not found Nov 02 17:29:34.847631 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:34.847577 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-nxnkz" podStartSLOduration=444.847557017 podStartE2EDuration="7m24.847557017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:31.940601794 +0000 UTC m=+31.879082990" watchObservedRunningTime="2026-11-02 17:29:34.847557017 +0000 UTC m=+34.786038429" Nov 02 17:29:34.850677 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:34.847979 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w8vvj"] Nov 02 17:29:34.861548 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:34.861500 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dmr89" event={"ID":"f7a7133f-8372-49a3-8194-15ed2159aa67","Type":"ContainerStarted","Data":"ddb7eb5478cec3d3e7aaabf6f53cc0935b6290be24fd9acc129f49703405246e"} Nov 02 17:29:34.905015 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:34.904948 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dmr89" podStartSLOduration=428.386206291 podStartE2EDuration="7m10.904927247s" totalImagesPullingTime="2.518720956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:34.90464544 +0000 UTC m=+34.843126638" watchObservedRunningTime="2026-11-02 17:29:34.904927247 +0000 UTC m=+34.843408449" Nov 02 17:29:35.047577 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:35.047497 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qjx25"] Nov 02 17:29:35.052649 ip-10-0-136-153 kubenswrapper[2576]: W1102 17:29:35.052568 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97af2a01_5609_4154_a892_5de8ae3e2b9a.slice/crio-39f2630e80b307bd7a346a66e784461be5ae36dad8ab9e62145a42b163f9c45d WatchSource:0}: Error finding container 39f2630e80b307bd7a346a66e784461be5ae36dad8ab9e62145a42b163f9c45d: Status 404 returned error can't find the container with id 39f2630e80b307bd7a346a66e784461be5ae36dad8ab9e62145a42b163f9c45d Nov 02 17:29:35.869292 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:35.869207 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qjx25" event={"ID":"97af2a01-5609-4154-a892-5de8ae3e2b9a","Type":"ContainerStarted","Data":"39f2630e80b307bd7a346a66e784461be5ae36dad8ab9e62145a42b163f9c45d"} Nov 02 17:29:35.871798 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:35.871770 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w8vvj" event={"ID":"b4de14e5-995e-4171-bb61-9f70e777688e","Type":"ContainerStarted","Data":"b127ce3c4bb20da36b450f6d58d6b07b125f3172440deda51dbb06f211cd4c5b"} Nov 02 17:29:36.877478 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:36.877433 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tszgp" event={"ID":"7e4dc8ce-2802-4864-aed5-affeefe41d6b","Type":"ContainerStarted","Data":"cf64d6d85efbfa52c4ba9c95a29f449a4685ac8cf8f950a0e41dc88ea9913122"} Nov 02 17:29:36.880481 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:36.880451 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/0.log" Nov 02 17:29:36.880606 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:36.880484 2576 generic.go:356] "Generic (PLEG): container finished" podID="5f9e8821-40a8-4012-b66a-813e80d856f2" containerID="55d9b9a7540458d13d18bf5c649fbc43469b610fa4b066231f825576c688fb2e" exitCode=255 Nov 02 17:29:36.880606 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:36.880534 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" event={"ID":"5f9e8821-40a8-4012-b66a-813e80d856f2","Type":"ContainerDied","Data":"55d9b9a7540458d13d18bf5c649fbc43469b610fa4b066231f825576c688fb2e"} Nov 02 17:29:36.881070 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:36.880809 2576 scope.go:122] "RemoveContainer" containerID="55d9b9a7540458d13d18bf5c649fbc43469b610fa4b066231f825576c688fb2e" Nov 02 17:29:36.929912 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:36.929012 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-tszgp" podStartSLOduration=404.122161877 podStartE2EDuration="6m48.928999985s" totalImagesPullingTime="4.806838108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:36.928327934 +0000 UTC m=+36.866809134" watchObservedRunningTime="2026-11-02 17:29:36.928999985 +0000 UTC m=+36.867481184" Nov 02 17:29:37.597055 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.597024 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fbrw7"] Nov 02 17:29:37.614588 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.614559 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:37.628016 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.627895 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fbrw7"] Nov 02 17:29:37.718779 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.718704 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 17:29:37.719157 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.719116 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fw7vt\"" Nov 02 17:29:37.720144 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.720078 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 17:29:37.753274 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.752780 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-data-volume\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:37.753274 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.752827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:37.753274 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.753009 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wn79\" (UniqueName: \"kubernetes.io/projected/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-api-access-5wn79\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:37.753274 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.753040 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:37.753274 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.753077 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-crio-socket\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:37.854606 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.853929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wn79\" (UniqueName: \"kubernetes.io/projected/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-api-access-5wn79\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:37.854606 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.853975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:37.854606 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.854016 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-crio-socket\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:37.854606 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.854072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-data-volume\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:37.854606 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.854096 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:37.854606 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:37.854162 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 17:29:37.854606 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:37.854244 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-rbac-proxy-cm podName:85cbfa4a-71cd-4b2c-8596-a484deeb15a5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:38.354224062 +0000 UTC m=+38.292705256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fbrw7" (UID: "85cbfa4a-71cd-4b2c-8596-a484deeb15a5") : configmap "kube-rbac-proxy" not found Nov 02 17:29:37.854606 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:37.854287 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 17:29:37.854606 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:37.854326 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls podName:85cbfa4a-71cd-4b2c-8596-a484deeb15a5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:38.35431651 +0000 UTC m=+38.292797694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fbrw7" (UID: "85cbfa4a-71cd-4b2c-8596-a484deeb15a5") : secret "insights-runtime-extractor-tls" not found Nov 02 17:29:37.854606 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.854380 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-crio-socket\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:37.855256 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.855165 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-data-volume\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:37.878339 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.878309 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wn79\" (UniqueName: \"kubernetes.io/projected/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-api-access-5wn79\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:37.887113 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.887090 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/1.log" Nov 02 17:29:37.887561 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.887543 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/0.log" Nov 02 17:29:37.887646 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.887580 2576 generic.go:356] "Generic (PLEG): container finished" podID="5f9e8821-40a8-4012-b66a-813e80d856f2" containerID="3d9ba56bcbd5f834961a29159d8acb53f61960759a915226ffb42f05b8a1ea33" exitCode=255 Nov 02 17:29:37.887835 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.887808 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" event={"ID":"5f9e8821-40a8-4012-b66a-813e80d856f2","Type":"ContainerDied","Data":"3d9ba56bcbd5f834961a29159d8acb53f61960759a915226ffb42f05b8a1ea33"} Nov 02 17:29:37.887976 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.887906 2576 scope.go:122] "RemoveContainer" containerID="55d9b9a7540458d13d18bf5c649fbc43469b610fa4b066231f825576c688fb2e" Nov 02 17:29:37.887976 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.887929 2576 scope.go:122] "RemoveContainer" containerID="3d9ba56bcbd5f834961a29159d8acb53f61960759a915226ffb42f05b8a1ea33" Nov 02 17:29:37.888153 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:37.888130 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-xqzqj_openshift-console-operator(5f9e8821-40a8-4012-b66a-813e80d856f2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" podUID="5f9e8821-40a8-4012-b66a-813e80d856f2" Nov 02 17:29:37.894901 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.894404 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4qfkf" event={"ID":"704c917e-c41f-47c3-9a1e-02730361fc8a","Type":"ContainerStarted","Data":"d881b95e3638b622669291d3289bdf8b7e213e8e3ca0fc812709a3fe854463d0"} Nov 02 17:29:37.896901 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.896875 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" event={"ID":"bc4a6db8-98e0-4387-8e64-1020978970c3","Type":"ContainerStarted","Data":"8662042c41316b6feddd4582a37f359ed6154b11db0c8fdc6eb54549fbd09633"} Nov 02 17:29:37.898896 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.898833 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" event={"ID":"a074a9ac-624c-4828-9785-c8f2aecb5827","Type":"ContainerStarted","Data":"9d96d086ad1f95c5550472add0b63ad439e1e3b5ce53a6d24bdbba7e89db53ed"} Nov 02 17:29:37.968373 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:37.967998 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4qfkf" podStartSLOduration=2.20863704 podStartE2EDuration="7.967979742s" totalImagesPullingTime="5.759342702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:37.966964598 +0000 UTC m=+37.905445796" watchObservedRunningTime="2026-11-02 17:29:37.967979742 +0000 UTC m=+37.906460943" Nov 02 17:29:38.014886 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:38.014127 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" podStartSLOduration=404.645838462 podStartE2EDuration="6m50.01410761s" totalImagesPullingTime="5.368269148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:38.013246058 +0000 UTC m=+37.951727258" watchObservedRunningTime="2026-11-02 17:29:38.01410761 +0000 UTC m=+37.952588830" Nov 02 17:29:38.061904 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:38.061021 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" podStartSLOduration=404.693957518 podStartE2EDuration="6m50.061001802s" totalImagesPullingTime="5.367044284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:38.059980417 +0000 UTC m=+37.998461616" watchObservedRunningTime="2026-11-02 17:29:38.061001802 +0000 UTC m=+37.999483002" Nov 02 17:29:38.360224 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:38.360182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:38.360377 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:38.360278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:38.360377 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.360323 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 17:29:38.360460 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.360387 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-rbac-proxy-cm podName:85cbfa4a-71cd-4b2c-8596-a484deeb15a5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:39.360367602 +0000 UTC m=+39.298848796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fbrw7" (UID: "85cbfa4a-71cd-4b2c-8596-a484deeb15a5") : configmap "kube-rbac-proxy" not found Nov 02 17:29:38.360460 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.360445 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 17:29:38.360526 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.360483 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls podName:85cbfa4a-71cd-4b2c-8596-a484deeb15a5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:39.360472969 +0000 UTC m=+39.298954149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fbrw7" (UID: "85cbfa4a-71cd-4b2c-8596-a484deeb15a5") : secret "insights-runtime-extractor-tls" not found Nov 02 17:29:38.663583 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:38.663504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:38.663732 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.663678 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 17:29:38.663732 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.663700 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8ffbdd9-xqf4t: secret "image-registry-tls" not found Nov 02 17:29:38.663839 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.663762 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls podName:27ea270b-6e55-4652-a1bf-1431a21c3d55 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.663743148 +0000 UTC m=+46.602224344 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls") pod "image-registry-6f8ffbdd9-xqf4t" (UID: "27ea270b-6e55-4652-a1bf-1431a21c3d55") : secret "image-registry-tls" not found Nov 02 17:29:38.765018 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:38.764983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:38.765018 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:38.765023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:38.765361 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:38.765071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7scpf\" (UID: \"72445bf5-a7a3-49dc-9b97-5589ae901032\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:38.765361 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:38.765105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:38.765361 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:38.765156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:38.765361 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.765171 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 17:29:38.765361 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:38.765196 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:38.765361 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:38.765217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:38.765361 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.765246 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls podName:42f9e736-411e-46a2-89d0-2a3870e2f95a nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.765225184 +0000 UTC m=+46.703706376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pthgk" (UID: "42f9e736-411e-46a2-89d0-2a3870e2f95a") : secret "cluster-monitoring-operator-tls" not found Nov 02 17:29:38.765361 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.765293 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 17:29:38.765361 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.765307 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78749f569b-zzhxm: secret "image-registry-tls" not found Nov 02 17:29:38.765361 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.765342 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls podName:7484d61b-5158-4729-b835-c0b372d8105b nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.765330764 +0000 UTC m=+46.703811941 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls") pod "image-registry-78749f569b-zzhxm" (UID: "7484d61b-5158-4729-b835-c0b372d8105b") : secret "image-registry-tls" not found Nov 02 17:29:38.765796 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.765389 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 17:29:38.765796 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.765417 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs podName:3d83bf78-e7b7-48bb-98be-8196e92fd320 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.765406672 +0000 UTC m=+46.703887897 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs") pod "router-default-79c77b5cd5-c6svb" (UID: "3d83bf78-e7b7-48bb-98be-8196e92fd320") : secret "router-metrics-certs-default" not found Nov 02 17:29:38.765796 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.765461 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 17:29:38.765796 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.765485 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert podName:72445bf5-a7a3-49dc-9b97-5589ae901032 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.765478817 +0000 UTC m=+46.703959993 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7scpf" (UID: "72445bf5-a7a3-49dc-9b97-5589ae901032") : secret "networking-console-plugin-cert" not found Nov 02 17:29:38.765796 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.765519 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 17:29:38.765796 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.765547 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls podName:f20b07b4-cd27-4410-abb2-c0bd50c01269 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.765531361 +0000 UTC m=+46.704012538 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dxkqn" (UID: "f20b07b4-cd27-4410-abb2-c0bd50c01269") : secret "samples-operator-tls" not found Nov 02 17:29:38.765796 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.765583 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 17:29:38.765796 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.765599 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls podName:a453427b-5025-4653-a930-ce2cf2f41cce nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.765594227 +0000 UTC m=+46.704075405 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls") pod "dns-default-w7zsl" (UID: "a453427b-5025-4653-a930-ce2cf2f41cce") : secret "dns-default-metrics-tls" not found Nov 02 17:29:38.765796 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.765650 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle podName:3d83bf78-e7b7-48bb-98be-8196e92fd320 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.765642419 +0000 UTC m=+46.704123640 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle") pod "router-default-79c77b5cd5-c6svb" (UID: "3d83bf78-e7b7-48bb-98be-8196e92fd320") : configmap references non-existent config key: service-ca.crt Nov 02 17:29:38.904431 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:38.904405 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/1.log" Nov 02 17:29:38.904887 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:38.904767 2576 scope.go:122] "RemoveContainer" containerID="3d9ba56bcbd5f834961a29159d8acb53f61960759a915226ffb42f05b8a1ea33" Nov 02 17:29:38.905031 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:38.905008 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-xqzqj_openshift-console-operator(5f9e8821-40a8-4012-b66a-813e80d856f2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" podUID="5f9e8821-40a8-4012-b66a-813e80d856f2" Nov 02 17:29:38.906384 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:38.906351 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5fvrw" event={"ID":"0eacac81-35d0-4ab1-bea4-25aaefa74c8b","Type":"ContainerStarted","Data":"f0e1f0668a5321d3d10cdeed134924a1c0dc27e9cee30d6141dfbd1c8dad3036"} Nov 02 17:29:38.908022 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:38.907997 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qjx25" event={"ID":"97af2a01-5609-4154-a892-5de8ae3e2b9a","Type":"ContainerStarted","Data":"e7f3356ef387ec50719c182a6bbd490eb458fd12a889f2c0c805d5ebb2a9017f"} Nov 02 17:29:38.984781 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:38.984684 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qjx25" podStartSLOduration=34.529828101 podStartE2EDuration="37.984670019s" totalImagesPullingTime="3.454841918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:38.984162877 +0000 UTC m=+38.922644075" watchObservedRunningTime="2026-11-02 17:29:38.984670019 +0000 UTC m=+38.923151215" Nov 02 17:29:39.024631 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:39.024542 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5fvrw" podStartSLOduration=397.706666927 podStartE2EDuration="6m44.024525158s" totalImagesPullingTime="6.317858231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:39.022712047 +0000 UTC m=+38.961193239" watchObservedRunningTime="2026-11-02 17:29:39.024525158 +0000 UTC m=+38.963006359" Nov 02 17:29:39.370465 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:39.370412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:39.370645 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:39.370561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:39.370645 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:39.370592 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 17:29:39.370754 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:39.370652 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 17:29:39.370754 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:39.370675 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls podName:85cbfa4a-71cd-4b2c-8596-a484deeb15a5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:41.370653415 +0000 UTC m=+41.309134606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fbrw7" (UID: "85cbfa4a-71cd-4b2c-8596-a484deeb15a5") : secret "insights-runtime-extractor-tls" not found Nov 02 17:29:39.370754 ip-10-0-136-153 kubenswrapper[2576]: E1102 17:29:39.370727 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-rbac-proxy-cm podName:85cbfa4a-71cd-4b2c-8596-a484deeb15a5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:41.37070166 +0000 UTC m=+41.309182858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fbrw7" (UID: "85cbfa4a-71cd-4b2c-8596-a484deeb15a5") : configmap "kube-rbac-proxy" not found Nov 02 17:29:39.912905 ip-10-0-136-153 kubenswrapper[2576]: I1102 17:29:39.912873 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:41.312361 ip-10-0-136-153 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 17:29:41.329242 ip-10-0-136-153 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 17:29:41.329411 ip-10-0-136-153 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 17:29:41.329458 ip-10-0-136-153 systemd[1]: kubelet.service: Consumed 2.912s CPU time. Nov 02 17:29:41.342489 ip-10-0-136-153 systemd[1]: Starting Kubernetes Kubelet... Nov 02 17:29:41.406907 ip-10-0-136-153 kubenswrapper[4860]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:29:41.406907 ip-10-0-136-153 kubenswrapper[4860]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 17:29:41.406907 ip-10-0-136-153 kubenswrapper[4860]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:29:41.406907 ip-10-0-136-153 kubenswrapper[4860]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:29:41.408956 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.408876 4860 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:41.408956 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.408955 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:41.409026 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.408960 4860 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:41.409026 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.408964 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:41.409026 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.408969 4860 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:41.409026 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.408974 4860 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:41.409026 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.408977 4860 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:41.409026 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.408981 4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:41.409026 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.408984 4860 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:41.409026 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.408987 4860 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:41.409026 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.408989 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:41.409026 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.408992 4860 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:41.409026 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.408995 4860 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:41.409026 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.408999 4860 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:41.409026 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409003 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:41.409026 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409006 4860 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:41.409026 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409009 4860 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:41.409026 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409012 4860 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:41.409026 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409015 4860 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:41.409026 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409018 4860 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:41.409026 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409020 4860 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409023 4860 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409026 4860 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409029 4860 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409032 4860 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409043 4860 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409047 4860 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409049 4860 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409052 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409054 4860 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409057 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409060 4860 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409063 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409065 4860 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409068 4860 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409070 4860 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409073 4860 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409075 4860 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409080 4860 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409083 4860 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:41.409476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409086 4860 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:41.409978 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409088 4860 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:41.409978 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409091 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:41.409978 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409093 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:41.409978 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409096 4860 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:41.409978 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409098 4860 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:41.409978 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409101 4860 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:41.409978 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409103 4860 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:41.409978 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409106 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:41.409978 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409109 4860 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:41.409978 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409111 4860 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:41.409978 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409114 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:41.409978 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409117 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:41.409978 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409119 4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:41.409978 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409123 4860 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:41.409978 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409125 4860 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:41.409978 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409128 4860 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:41.409978 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409131 4860 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:41.409978 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409140 4860 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:41.409978 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409143 4860 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409146 4860 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409148 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409151 4860 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409153 4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409156 4860 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409158 4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409161 4860 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409164 4860 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409166 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409169 4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409177 4860 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409179 4860 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409182 4860 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409184 4860 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409187 4860 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409189 4860 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409192 4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409195 4860 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409198 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:41.410529 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409200 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409203 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409206 4860 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409210 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409213 4860 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409216 4860 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409218 4860 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409221 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409223 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409226 4860 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409229 4860 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409231 4860 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409239 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409242 4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409245 4860 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409247 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409250 4860 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409253 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409255 4860 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409258 4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:41.411054 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409261 4860 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:41.411537 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409263 4860 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:41.411537 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409266 4860 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:41.411537 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409270 4860 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:41.411537 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409272 4860 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:41.411537 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409275 4860 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:41.411537 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.409277 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:41.411537 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410024 4860 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:41.411537 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410031 4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:41.411537 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410034 4860 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:41.411537 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410037 4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:41.411537 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410040 4860 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:41.411537 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410043 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:41.411537 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410046 4860 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:41.411537 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410048 4860 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:41.411537 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410051 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:41.411537 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410054 4860 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:41.411537 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410056 4860 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:41.411537 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410059 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:41.411537 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410062 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410064 4860 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410067 4860 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410070 4860 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410072 4860 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410079 4860 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410085 4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410088 4860 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410091 4860 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410093 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410096 4860 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410099 4860 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410101 4860 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410104 4860 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410106 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410109 4860 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410112 4860 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410115 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410117 4860 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410120 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:41.412006 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410123 4860 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:41.412483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410125 4860 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:41.412483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410128 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:41.412483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410130 4860 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:41.412483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410133 4860 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:41.412483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410136 4860 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:41.412483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410138 4860 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:41.412483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410141 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:41.412483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410144 4860 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:41.412483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410146 4860 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:41.412483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410149 4860 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:41.412483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410151 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:41.412483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410154 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:41.412483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410157 4860 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:41.412483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410160 4860 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:41.412483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410162 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:41.412483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410165 4860 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:41.412483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410169 4860 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:41.412483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410172 4860 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:41.412483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410207 4860 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410211 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410213 4860 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410216 4860 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410219 4860 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410221 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410224 4860 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410227 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410229 4860 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410232 4860 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410235 4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410238 4860 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410240 4860 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410243 4860 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410245 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410248 4860 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410251 4860 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410254 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410256 4860 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410259 4860 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:41.412980 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410262 4860 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:41.413506 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410265 4860 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:41.413506 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410269 4860 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:41.413506 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410273 4860 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:41.413506 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410277 4860 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:41.413506 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410280 4860 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:41.413506 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410283 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:41.413506 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410285 4860 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:41.413506 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410289 4860 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:41.413506 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410294 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:41.413506 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410299 4860 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:41.413506 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410302 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:41.413506 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410305 4860 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:41.413506 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410308 4860 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:41.413506 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410339 4860 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:41.413506 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410343 4860 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:41.413506 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410346 4860 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:41.413506 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410353 4860 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:41.413506 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410357 4860 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:41.413506 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410359 4860 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410362 4860 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410365 4860 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410367 4860 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410370 4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410373 4860 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410375 4860 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410378 4860 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410380 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410383 4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410386 4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410388 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410391 4860 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410393 4860 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410396 4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410398 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410920 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410927 4860 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410930 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410934 4860 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:41.413997 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410937 4860 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410940 4860 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410943 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410948 4860 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410951 4860 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410954 4860 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410957 4860 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410960 4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410971 4860 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410974 4860 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410977 4860 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410980 4860 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410983 4860 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410986 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410988 4860 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410991 4860 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410994 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410997 4860 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.410999 4860 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411002 4860 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:41.414479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411005 4860 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411007 4860 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411010 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411013 4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411015 4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411018 4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411020 4860 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411023 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411025 4860 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411028 4860 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411030 4860 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411033 4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411036 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411038 4860 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411041 4860 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411045 4860 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411048 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411063 4860 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411067 4860 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411071 4860 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:41.415089 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411076 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411079 4860 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411081 4860 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411084 4860 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411088 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411091 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411093 4860 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411096 4860 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411098 4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411101 4860 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411104 4860 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411106 4860 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411109 4860 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411112 4860 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411115 4860 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411117 4860 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411120 4860 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411122 4860 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411125 4860 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411128 4860 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:41.415943 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411130 4860 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411133 4860 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411135 4860 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411138 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411141 4860 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411143 4860 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411145 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411149 4860 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411151 4860 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411154 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411156 4860 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411159 4860 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411163 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411166 4860 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411168 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411171 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411173 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411176 4860 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411178 4860 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411181 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:41.416487 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411183 4860 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:41.416998 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411186 4860 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:41.416998 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411188 4860 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:41.416998 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411191 4860 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:41.416998 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411193 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:41.416998 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411196 4860 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:41.416998 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411198 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:41.416998 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411201 4860 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:41.416998 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411203 4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:41.416998 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411206 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:41.416998 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411208 4860 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:41.416998 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411211 4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:41.416998 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411213 4860 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:41.416998 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411216 4860 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:41.416998 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411218 4860 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:41.416998 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411221 4860 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:41.416998 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411223 4860 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:41.416998 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411227 4860 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:41.416998 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411230 4860 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:41.416998 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411233 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411238 4860 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.411242 4860 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411335 4860 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411345 4860 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411356 4860 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411364 4860 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411368 4860 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411372 4860 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411376 4860 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411380 4860 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411384 4860 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411387 4860 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411390 4860 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411394 4860 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411397 4860 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411400 4860 flags.go:64] FLAG: --cgroup-root="" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411403 4860 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411406 4860 flags.go:64] FLAG: --client-ca-file="" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411409 4860 flags.go:64] FLAG: --cloud-provider="external" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411412 4860 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411417 4860 flags.go:64] FLAG: --cluster-domain="" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411420 4860 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 17:29:41.417453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411423 4860 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411426 4860 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411430 4860 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411434 4860 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411437 4860 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411440 4860 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411444 4860 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411447 4860 flags.go:64] FLAG: --contention-profiling="false" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411450 4860 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411453 4860 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411456 4860 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411459 4860 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411463 4860 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411467 4860 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411471 4860 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411474 4860 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411478 4860 flags.go:64] FLAG: --enable-server="true" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411481 4860 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411485 4860 flags.go:64] FLAG: --event-burst="100" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411488 4860 flags.go:64] FLAG: --event-qps="50" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411491 4860 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411494 4860 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411497 4860 flags.go:64] FLAG: --eviction-hard="" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411502 4860 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 17:29:41.418017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411505 4860 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411508 4860 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411511 4860 flags.go:64] FLAG: --eviction-soft="" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411514 4860 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411517 4860 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411520 4860 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411523 4860 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411526 4860 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411529 4860 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411532 4860 flags.go:64] FLAG: --feature-gates="" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411539 4860 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411543 4860 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411546 4860 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411549 4860 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411552 4860 flags.go:64] FLAG: --healthz-port="10248" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411556 4860 flags.go:64] FLAG: --help="false" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411558 4860 flags.go:64] FLAG: --hostname-override="ip-10-0-136-153.ec2.internal" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411562 4860 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411565 4860 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411568 4860 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411571 4860 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411575 4860 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411579 4860 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 17:29:41.418613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411582 4860 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411585 4860 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411589 4860 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411593 4860 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411597 4860 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411600 4860 flags.go:64] FLAG: --kube-reserved="" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411603 4860 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411606 4860 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411609 4860 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411612 4860 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411615 4860 flags.go:64] FLAG: --lock-file="" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411618 4860 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411621 4860 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411624 4860 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411629 4860 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411633 4860 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411636 4860 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411639 4860 flags.go:64] FLAG: --logging-format="text" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411642 4860 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411645 4860 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411648 4860 flags.go:64] FLAG: --manifest-url="" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411651 4860 flags.go:64] FLAG: --manifest-url-header="" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411656 4860 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411659 4860 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411663 4860 flags.go:64] FLAG: --max-pods="110" Nov 02 17:29:41.419238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411666 4860 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411669 4860 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411672 4860 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411675 4860 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411678 4860 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411681 4860 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411686 4860 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411693 4860 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411696 4860 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411699 4860 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411704 4860 flags.go:64] FLAG: --pod-cidr="" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411707 4860 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411711 4860 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411714 4860 flags.go:64] FLAG: --pods-per-core="0" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411717 4860 flags.go:64] FLAG: --port="10250" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411720 4860 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411723 4860 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dc428c8daadfdbe1" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411726 4860 flags.go:64] FLAG: --qos-reserved="" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411730 4860 flags.go:64] FLAG: --read-only-port="10255" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411733 4860 flags.go:64] FLAG: --register-node="true" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411736 4860 flags.go:64] FLAG: --register-with-taints="" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411740 4860 flags.go:64] FLAG: --registry-burst="10" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411743 4860 flags.go:64] FLAG: --registry-qps="5" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411746 4860 flags.go:64] FLAG: --reserved-cpus="" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411749 4860 flags.go:64] FLAG: --reserved-memory="" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411753 4860 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 17:29:41.419964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411756 4860 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411759 4860 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411762 4860 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411765 4860 flags.go:64] FLAG: --runonce="false" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411768 4860 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411771 4860 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411774 4860 flags.go:64] FLAG: --seccomp-default="false" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411777 4860 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411780 4860 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411783 4860 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411786 4860 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411790 4860 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411794 4860 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411797 4860 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411800 4860 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411803 4860 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411807 4860 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411811 4860 flags.go:64] FLAG: --system-cgroups="" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411814 4860 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411819 4860 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411822 4860 flags.go:64] FLAG: --tls-cert-file="" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411825 4860 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411830 4860 flags.go:64] FLAG: --tls-min-version="" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411833 4860 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411836 4860 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 17:29:41.420611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411839 4860 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411842 4860 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411856 4860 flags.go:64] FLAG: --v="2" Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411861 4860 flags.go:64] FLAG: --version="false" Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411865 4860 flags.go:64] FLAG: --vmodule="" Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411869 4860 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.411872 4860 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412008 4860 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412012 4860 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412016 4860 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412020 4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412023 4860 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412026 4860 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412028 4860 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412031 4860 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412034 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412036 4860 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412039 4860 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412042 4860 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412047 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412049 4860 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:41.421245 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412052 4860 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412055 4860 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412059 4860 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412062 4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412065 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412068 4860 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412070 4860 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412073 4860 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412075 4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412078 4860 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412080 4860 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412083 4860 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412085 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412088 4860 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412090 4860 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412093 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412095 4860 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412098 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412101 4860 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412103 4860 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:41.421976 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412106 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412108 4860 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412111 4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412114 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412116 4860 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412119 4860 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412121 4860 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412124 4860 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412127 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412129 4860 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412134 4860 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412137 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412139 4860 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412142 4860 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412146 4860 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412148 4860 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412151 4860 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412154 4860 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412157 4860 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412159 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:41.422665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412161 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:41.423328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412164 4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:41.423328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412167 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:41.423328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412169 4860 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:41.423328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412172 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:41.423328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412174 4860 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:41.423328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412177 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:41.423328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412179 4860 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:41.423328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412182 4860 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:41.423328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412185 4860 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:41.423328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412187 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:41.423328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412190 4860 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:41.423328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412192 4860 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:41.423328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412195 4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:41.423328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412197 4860 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:41.423328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412200 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:41.423328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412202 4860 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:41.423328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412205 4860 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:41.423328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412207 4860 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:41.423328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412210 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412212 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412215 4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412219 4860 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412222 4860 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412224 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412227 4860 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412231 4860 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412234 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412236 4860 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412239 4860 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412242 4860 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412245 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412247 4860 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412250 4860 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412253 4860 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412255 4860 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412258 4860 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412260 4860 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412263 4860 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:41.423801 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412265 4860 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:41.424339 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412269 4860 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:41.424339 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412274 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:41.424339 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412277 4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:41.424339 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412279 4860 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:41.424339 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412282 4860 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:41.424339 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412285 4860 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:41.424339 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412288 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:41.424339 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412290 4860 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:41.424339 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412293 4860 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:41.424339 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412295 4860 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:41.424339 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412298 4860 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:41.424339 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412300 4860 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:41.424339 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.412325 4860 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:29:41.424339 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412364 4860 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:41.424339 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412369 4860 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:41.424339 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412372 4860 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:41.424817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412375 4860 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:41.424817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412378 4860 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:41.424817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412381 4860 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:41.424817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412385 4860 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:41.424817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412387 4860 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:41.424817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412391 4860 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:41.424817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412395 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:41.424817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412399 4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:41.424817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412402 4860 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:41.424817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412406 4860 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:41.424817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412409 4860 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:41.424817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412411 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:41.424817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412414 4860 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:41.424817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412416 4860 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:41.424817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412419 4860 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:41.424817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412422 4860 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:41.424817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412424 4860 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:41.424817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412427 4860 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:41.424817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412430 4860 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412433 4860 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412438 4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412441 4860 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412444 4860 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412447 4860 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412449 4860 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412453 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412455 4860 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412458 4860 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412461 4860 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412466 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412470 4860 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412473 4860 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412475 4860 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412478 4860 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412481 4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412485 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412488 4860 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412492 4860 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:41.425328 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412495 4860 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412497 4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412501 4860 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412503 4860 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412506 4860 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412508 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412511 4860 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412513 4860 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412516 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412518 4860 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412521 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412524 4860 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412526 4860 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412529 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412531 4860 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412534 4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412536 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412539 4860 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412541 4860 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412544 4860 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:41.425869 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412547 4860 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412549 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412552 4860 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412554 4860 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412558 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412560 4860 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412563 4860 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412566 4860 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412568 4860 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412572 4860 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412574 4860 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412577 4860 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412580 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412582 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412585 4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412588 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412591 4860 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412593 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412596 4860 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412598 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:41.426342 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412601 4860 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:41.426817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412604 4860 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:41.426817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412606 4860 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:41.426817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412609 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:41.426817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412611 4860 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:41.426817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412614 4860 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:41.426817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412617 4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:41.426817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412619 4860 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:41.426817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412622 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:41.426817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412624 4860 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:41.426817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412627 4860 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:41.426817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412630 4860 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:41.426817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412632 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:41.426817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412635 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:41.426817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412637 4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:41.426817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412640 4860 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:41.426817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412643 4860 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:41.426817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412646 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:41.426817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412648 4860 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:41.426817 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412651 4860 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:41.427416 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412653 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:41.427416 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412657 4860 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:41.427416 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412659 4860 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:41.427416 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412662 4860 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:41.427416 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.412665 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:41.427416 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.417413 4860 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 17:29:41.427416 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.417436 4860 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 17:29:41.427416 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.417455 4860 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 17:29:41.427416 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.417461 4860 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 17:29:41.427416 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417518 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:41.427416 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417523 4860 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:41.427416 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417526 4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:41.427416 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417529 4860 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:41.427416 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417532 4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:41.427416 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417535 4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:41.427416 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417540 4860 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:41.427416 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417544 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:41.427416 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417547 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:41.427416 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417550 4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:41.427901 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417554 4860 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:41.427901 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417558 4860 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:41.427901 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417561 4860 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:41.427901 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417564 4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:41.427901 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417567 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:41.427901 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417569 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:41.427901 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417573 4860 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:41.427901 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417576 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:41.427901 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417579 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:41.427901 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417582 4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:41.427901 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417585 4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:41.427901 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417588 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:41.427901 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417591 4860 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:41.427901 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417594 4860 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:41.427901 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417597 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:41.427901 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417600 4860 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:41.427901 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417602 4860 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:41.427901 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417605 4860 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:41.427901 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417608 4860 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:41.428354 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417611 4860 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:41.428354 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417613 4860 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:41.428354 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417616 4860 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:41.428354 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417619 4860 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:41.428354 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417621 4860 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:41.428354 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417624 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:41.428354 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417627 4860 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:41.428354 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417629 4860 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:41.428354 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417632 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:41.428354 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417634 4860 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:41.428354 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417637 4860 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:41.428354 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417640 4860 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:41.428354 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417642 4860 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:41.428354 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417645 4860 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:41.428354 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417647 4860 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:41.428354 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417650 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:41.428354 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417653 4860 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:41.428354 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417656 4860 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:41.428354 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417659 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:41.428942 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417662 4860 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:41.428942 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417664 4860 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:41.428942 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417668 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:41.428942 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417671 4860 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:41.428942 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417673 4860 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:41.428942 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417676 4860 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:41.428942 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417679 4860 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:41.428942 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417681 4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:41.428942 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417684 4860 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:41.428942 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417686 4860 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:41.428942 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417689 4860 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:41.428942 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417691 4860 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:41.428942 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417694 4860 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:41.428942 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417696 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:41.428942 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417699 4860 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:41.428942 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417702 4860 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:41.428942 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417704 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:41.428942 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417707 4860 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:41.428942 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417709 4860 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417712 4860 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417715 4860 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417718 4860 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417721 4860 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417724 4860 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417727 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417729 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417732 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417734 4860 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417737 4860 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417739 4860 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417742 4860 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417749 4860 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417752 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417754 4860 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417758 4860 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417760 4860 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417763 4860 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417766 4860 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:41.429665 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417768 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:41.430179 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417771 4860 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:41.430179 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417774 4860 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:41.430179 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417776 4860 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:41.430179 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417779 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:41.430179 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417781 4860 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:41.430179 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417784 4860 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:41.430179 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417787 4860 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:41.430179 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417789 4860 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:41.430179 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417792 4860 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:41.430179 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417794 4860 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:41.430179 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417797 4860 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:41.430179 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417800 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:41.430179 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417802 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:41.430179 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417805 4860 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:41.430179 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417807 4860 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:41.430179 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417810 4860 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:41.430179 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417813 4860 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:41.430179 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417816 4860 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:41.430179 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.417818 4860 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:41.430658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.417867 4860 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:29:41.430658 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418012 4860 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:41.430658 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418017 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:41.430658 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418021 4860 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:41.430658 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418024 4860 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:41.430658 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418027 4860 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:41.430658 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418031 4860 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:41.430658 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418034 4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:41.430658 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418037 4860 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:41.430658 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418039 4860 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:41.430658 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418042 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:41.430658 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418045 4860 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:41.430658 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418048 4860 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:41.430658 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418051 4860 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:41.430658 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418053 4860 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:41.430658 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418056 4860 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:41.430658 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418059 4860 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:41.431182 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418061 4860 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:41.431182 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418064 4860 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:41.431182 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418067 4860 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:41.431182 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418070 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:41.431182 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418072 4860 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:41.431182 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418075 4860 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:41.431182 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418077 4860 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:41.431182 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418080 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:41.431182 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418083 4860 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:41.431182 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418085 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:41.431182 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418088 4860 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:41.431182 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418091 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:41.431182 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418093 4860 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:41.431182 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418096 4860 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:41.431182 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418098 4860 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:41.431182 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418101 4860 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:41.431182 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418104 4860 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:41.431182 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418106 4860 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:41.431182 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418109 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418111 4860 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418114 4860 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418117 4860 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418121 4860 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418123 4860 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418126 4860 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418129 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418131 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418134 4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418136 4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418139 4860 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418141 4860 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418144 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418147 4860 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418149 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418152 4860 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418154 4860 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418157 4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418159 4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:41.431629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418162 4860 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418165 4860 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418168 4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418170 4860 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418173 4860 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418175 4860 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418178 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418180 4860 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418183 4860 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418186 4860 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418188 4860 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418191 4860 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418193 4860 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418196 4860 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418198 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418201 4860 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418204 4860 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418209 4860 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418212 4860 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418215 4860 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:41.432117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418218 4860 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:41.432583 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418220 4860 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:41.432583 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418223 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:41.432583 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418226 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:41.432583 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418228 4860 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:41.432583 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418231 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:41.432583 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418234 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:41.432583 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418236 4860 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:41.432583 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418239 4860 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:41.432583 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418241 4860 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:41.432583 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418244 4860 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:41.432583 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418246 4860 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:41.432583 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418249 4860 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:41.432583 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418251 4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:41.432583 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418254 4860 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:41.432583 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418256 4860 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:41.432583 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418259 4860 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:41.432583 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418261 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:41.432583 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418264 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:41.432583 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418267 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:41.433092 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418270 4860 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:41.433092 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418272 4860 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:41.433092 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418275 4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:41.433092 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418279 4860 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:41.433092 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418282 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:41.433092 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418285 4860 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:41.433092 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418288 4860 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:41.433092 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418291 4860 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:41.433092 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418296 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:41.433092 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418299 4860 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:41.433092 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418302 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:41.433092 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418305 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:41.433092 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.418329 4860 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:29:41.433092 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418359 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:29:41.433092 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418363 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:29:41.433458 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418367 4860 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:29:41.433458 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418370 4860 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:29:41.433458 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418373 4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:29:41.433458 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418377 4860 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:29:41.433458 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418381 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:29:41.433458 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418385 4860 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:29:41.433458 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418388 4860 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:29:41.433458 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418391 4860 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:29:41.433458 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418394 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:29:41.433458 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418398 4860 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:29:41.433458 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418400 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:29:41.433458 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418403 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:29:41.433458 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418406 4860 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:29:41.433458 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418408 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:29:41.433458 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418411 4860 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:29:41.433458 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418413 4860 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:29:41.433458 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418416 4860 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:29:41.433458 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418419 4860 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:29:41.433458 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418421 4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:29:41.433974 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418424 4860 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:29:41.433974 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418426 4860 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:29:41.433974 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418429 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:29:41.433974 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418432 4860 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:29:41.433974 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418435 4860 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:29:41.433974 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418437 4860 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:29:41.433974 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418440 4860 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:29:41.433974 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418444 4860 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:29:41.433974 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418447 4860 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:29:41.433974 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418450 4860 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:29:41.433974 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418453 4860 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:29:41.433974 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418455 4860 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:29:41.433974 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418458 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:29:41.433974 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418461 4860 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:29:41.433974 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418465 4860 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:29:41.433974 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418468 4860 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:29:41.433974 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418470 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:29:41.433974 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418473 4860 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:29:41.433974 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418476 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418478 4860 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418481 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418483 4860 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418487 4860 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418489 4860 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418492 4860 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418494 4860 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418497 4860 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418499 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418502 4860 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418505 4860 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418507 4860 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418510 4860 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418512 4860 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418515 4860 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418517 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418520 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418522 4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418525 4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:29:41.434445 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418527 4860 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418531 4860 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418534 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418537 4860 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418540 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418543 4860 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418545 4860 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418548 4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418552 4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418555 4860 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418557 4860 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418560 4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418563 4860 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418565 4860 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418568 4860 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418570 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418573 4860 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418576 4860 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418579 4860 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418581 4860 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:29:41.434970 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418584 4860 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:29:41.435447 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418587 4860 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:29:41.435447 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418589 4860 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:29:41.435447 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418592 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:29:41.435447 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418594 4860 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:29:41.435447 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418597 4860 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:29:41.435447 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418601 4860 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:29:41.435447 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418605 4860 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:29:41.435447 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418608 4860 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:29:41.435447 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418610 4860 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:29:41.435447 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418613 4860 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:29:41.435447 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418616 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:29:41.435447 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418619 4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:29:41.435447 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418623 4860 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:29:41.435447 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418626 4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:29:41.435447 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418629 4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:29:41.435447 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418631 4860 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:29:41.435447 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418634 4860 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:29:41.435447 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418637 4860 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:29:41.435447 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418640 4860 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:29:41.435972 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418643 4860 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:29:41.435972 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418647 4860 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:29:41.435972 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418649 4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:29:41.435972 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418652 4860 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:29:41.435972 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418655 4860 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:29:41.435972 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418658 4860 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:29:41.435972 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:41.418660 4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:29:41.435972 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.418714 4860 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 17:29:41.435972 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.419692 4860 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 17:29:41.435972 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.419748 4860 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 17:29:41.435972 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.419935 4860 server.go:1014] "Starting client certificate rotation" Nov 02 17:29:41.435972 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.420073 4860 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:29:41.435972 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.420126 4860 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:24:01 +0000 UTC" deadline="2028-04-29 17:53:23.517692703 +0000 UTC" Nov 02 17:29:41.435972 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.420154 4860 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13056h23m42.097540743s" Nov 02 17:29:41.435972 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.420379 4860 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:29:41.435972 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.421429 4860 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:29:41.436364 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.422684 4860 log.go:25] "Validated CRI v1 runtime API" Nov 02 17:29:41.436364 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.424324 4860 log.go:25] "Validated CRI v1 image API" Nov 02 17:29:41.436364 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.425111 4860 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 17:29:41.436364 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.426927 4860 fs.go:135] Filesystem UUIDs: map[5924b1c7-711f-43f8-ab68-87abd407560c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f988989c-f35d-46ed-b30f-7d7057b62aa5:/dev/nvme0n1p3] Nov 02 17:29:41.436492 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.426947 4860 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19e5c90135094488bf4817a4198f7e14939531ab670190f348241b436254c4f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19e5c90135094488bf4817a4198f7e14939531ab670190f348241b436254c4f1/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1bdf615d401fa21d9ac512784085320be2cb538e86cc41992d168f339019b37e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1bdf615d401fa21d9ac512784085320be2cb538e86cc41992d168f339019b37e/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f82b9b75a1d021de115fd0758e953cb5e873d00a38ed422f1c672362623e0b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f82b9b75a1d021de115fd0758e953cb5e873d00a38ed422f1c672362623e0b0/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39f2630e80b307bd7a346a66e784461be5ae36dad8ab9e62145a42b163f9c45d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39f2630e80b307bd7a346a66e784461be5ae36dad8ab9e62145a42b163f9c45d/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4072df9c777140fe13d8e46d3c3c46ce4d4842f2cdb8f0e9ad3db97af69c50a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4072df9c777140fe13d8e46d3c3c46ce4d4842f2cdb8f0e9ad3db97af69c50a3/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/533e0dcf6ba7212004f611f9b2995fc59b1eb2eff0cb758582001a69e7cddbad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/533e0dcf6ba7212004f611f9b2995fc59b1eb2eff0cb758582001a69e7cddbad/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5db7566245c6eb0c1454d809ae7c7ffe49dd9a6b0aeb3b2b66136f88fb214087/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5db7566245c6eb0c1454d809ae7c7ffe49dd9a6b0aeb3b2b66136f88fb214087/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61880416d13b3c700b791ab52cfdc2ebbc6aa31c487736166dc2bbe25b665e32/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61880416d13b3c700b791ab52cfdc2ebbc6aa31c487736166dc2bbe25b665e32/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69bf746fbe676b165b8dc071761c74a42b201c4e3db51d6f2afcd1b14ea86d4e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69bf746fbe676b165b8dc071761c74a42b201c4e3db51d6f2afcd1b14ea86d4e/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/732370e3f644cd5f750ef139deb4731284af5b0051284347014685a23ed3b579/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/732370e3f644cd5f750ef139deb4731284af5b0051284347014685a23ed3b579/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74924f8e0f80598f0ae2d2226505477ff26bbd3e35ef383ffd97234d57604a1e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74924f8e0f80598f0ae2d2226505477ff26bbd3e35ef383ffd97234d57604a1e/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c38e5e87ea0feaac1a2bf835454b3c0968e9442f8fba1d8225f9b3565583717/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c38e5e87ea0feaac1a2bf835454b3c0968e9442f8fba1d8225f9b3565583717/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afadebb9c02b6aa05450a3103a4757f749c4382bb8afebb0b9f81979d07c49f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afadebb9c02b6aa05450a3103a4757f749c4382bb8afebb0b9f81979d07c49f7/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afb720b260f19c89ec5d6fd564551c31801043e60c91fe6cd422f21f3a143d88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afb720b260f19c89ec5d6fd564551c31801043e60c91fe6cd422f21f3a143d88/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b127ce3c4bb20da36b450f6d58d6b07b125f3172440deda51dbb06f211cd4c5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b127ce3c4bb20da36b450f6d58d6b07b125f3172440deda51dbb06f211cd4c5b/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c35565f0de21fb64553bae63d86fa9635c3e543017dd84bf2ba17486215cda7a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c35565f0de21fb64553bae63d86fa9635c3e543017dd84bf2ba17486215cda7a/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd8bb8092f4aed3a7e68285f1a30d1bcaae6ae0c5568d26f8f9bb3a5643cdd85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd8bb8092f4aed3a7e68285f1a30d1bcaae6ae0c5568d26f8f9bb3a5643cdd85/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5dd2759fbe8d71f6c45700e5a83443fcb3baac4c517e585e0e7245c0e956460/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5dd2759fbe8d71f6c45700e5a83443fcb3baac4c517e585e0e7245c0e956460/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de2b882663fbc211a07a1f8327255abab18faeefc69d5113bffae1f9a25149c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de2b882663fbc211a07a1f8327255abab18faeefc69d5113bffae1f9a25149c8/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dfff4b53674ec62275631a314edb707b7e7f3be4efccafa74bd017d5f347212e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dfff4b53674ec62275631a314edb707b7e7f3be4efccafa74bd017d5f347212e/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5e336b15af15f9570964afe8da5675163e1868f984595421ce2c9f2a07cb7e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5e336b15af15f9570964afe8da5675163e1868f984595421ce2c9f2a07cb7e2/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6d4af786dd97bc313a2a112a07af8b16a79f2f3d4ef347307b54d3916c868c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6d4af786dd97bc313a2a112a07af8b16a79f2f3d4ef347307b54d3916c868c6/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0eacac81-35d0-4ab1-bea4-25aaefa74c8b/volumes/kubernetes.io~projected/kube-api-access-njp5v:{mountpoint:/var/lib/kubelet/pods/0eacac81-35d0-4ab1-bea4-25aaefa74c8b/volumes/kubernetes.io~projected/kube-api-access-njp5v major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27ea270b-6e55-4652-a1bf-1431a21c3d55/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/27ea270b-6e55-4652-a1bf-1431a21c3d55/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27ea270b-6e55-4652-a1bf-1431a21c3d55/volumes/kubernetes.io~projected/kube-api-access-rpdh7:{mountpoint:/var/lib/kubelet/pods/27ea270b-6e55-4652-a1bf-1431a21c3d55/volumes/kubernetes.io~projected/kube-api-access-rpdh7 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27ea270b-6e55-4652-a1bf-1431a21c3d55/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/27ea270b-6e55-4652-a1bf-1431a21c3d55/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27ea270b-6e55-4652-a1bf-1431a21c3d55/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/27ea270b-6e55-4652-a1bf-1431a21c3d55/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35669b5e-5c97-45ef-8868-4cff3fdfc07a/volumes/kubernetes.io~projected/kube-api-access-lrvfr:{mountpoint:/var/lib/kubelet/pods/35669b5e-5c97-45ef-8868-4cff3fdfc07a/volumes/kubernetes.io~projected/kube-api-access-lrvfr major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38e5c802-99d4-4234-9e64-dee8e19b055a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/38e5c802-99d4-4234-9e64-dee8e19b055a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38e5c802-99d4-4234-9e64-dee8e19b055a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/38e5c802-99d4-4234-9e64-dee8e19b055a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38e5c802-99d4-4234-9e64-dee8e19b055a/volumes/kubernetes.io~projected/kube-api-access-jphfq:{mountpoint:/var/lib/kubelet/pods/38e5c802-99d4-4234-9e64-dee8e19b055a/volumes/kubernetes.io~projected/kube-api-access-jphfq major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d83bf78-e7b7-48bb-98be-8196e92fd320/volumes/kubernetes.io~projected/kube-api-access-mm8pr:{mountpoint:/var/lib/kubelet/pods/3d83bf78-e7b7-48bb-98be-8196e92fd320/volumes/kubernetes.io~projected/kube-api-access-mm8pr major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d83bf78-e7b7-48bb-98be-8196e92fd320/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/3d83bf78-e7b7-48bb-98be-8196e92fd320/volumes/kubernetes.io~secret/default-certificate major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d83bf78-e7b7-48bb-98be-8196e92fd320/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/3d83bf78-e7b7-48bb-98be-8196e92fd320/volumes/kubernetes.io~secret/stats-auth major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42f9e736-411e-46a2-89d0-2a3870e2f95a/volumes/kubernetes.io~projected/kube-api-access-8jf4x:{mountpoint:/var/lib/kubelet/pods/42f9e736-411e-46a2-89d0-2a3870e2f95a/volumes/kubernetes.io~projected/kube-api-access-8jf4x major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46d2456f-62b2-4aa5-b172-1c28ab7615ac/volumes/kubernetes.io~projected/kube-api-access-wqrgs:{mountpoint:/var/lib/kubelet/pods/46d2456f-62b2-4aa5-b172-1c28ab7615ac/volumes/kubernetes.io~projected/kube-api-access-wqrgs major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b533156-8ec1-4994-9bf6-75e4c41c3c6a/volumes/kubernetes.io~projected/kube-api-access-zdlkh:{mountpoint:/var/lib/kubelet/pods/4b533156-8ec1-4994-9bf6-75e4c41c3c6a/volumes/kubernetes.io~projected/kube-api-access-zdlkh major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b5c29e0-6bfd-4f43-867a-c460dc227fbc/volumes/kubernetes.io~projected/kube-api-access-n8psf:{mountpoint:/var/lib/kubelet/pods/4b5c29e0-6bfd-4f43-867a-c460dc227fbc/volumes/kubernetes.io~projected/kube-api-access-n8psf major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f9e8821-40a8-4012-b66a-813e80d856f2/volumes/kubernetes.io~projected/kube-api-access-spg94:{mountpoint:/var/lib/kubelet/pods/5f9e8821-40a8-4012-b66a-813e80d856f2/volumes/kubernetes.io~projected/kube-api-access-spg94 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f9e8821-40a8-4012-b66a-813e80d856f2/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5f9e8821-40a8-4012-b66a-813e80d856f2/volumes/kubernetes.io~secret/serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/704c917e-c41f-47c3-9a1e-02730361fc8a/volumes/kubernetes.io~projected/kube-api-access-jd9gl:{mountpoint:/var/lib/kubelet/pods/704c917e-c41f-47c3-9a1e-02730361fc8a/volumes/kubernetes.io~projected/kube-api-access-jd9gl major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/704c917e-c41f-47c3-9a1e-02730361fc8a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/704c917e-c41f-47c3-9a1e-02730361fc8a/volumes/kubernetes.io~secret/cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7484d61b-5158-4729-b835-c0b372d8105b/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/7484d61b-5158-4729-b835-c0b372d8105b/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7484d61b-5158-4729-b835-c0b372d8105b/volumes/kubernetes.io~projected/kube-api-access-2b4dt:{mountpoint:/var/lib/kubelet/pods/7484d61b-5158-4729-b835-c0b372d8105b/volumes/kubernetes.io~projected/kube-api-access-2b4dt major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7484d61b-5158-4729-b835-c0b372d8105b/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/7484d61b-5158-4729-b835-c0b372d8105b/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7484d61b-5158-4729-b835-c0b372d8105b/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/7484d61b-5158-4729-b835-c0b372d8105b/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e4dc8ce-2802-4864-aed5-affeefe41d6b/volumes/kubernetes.io~projected/kube-api-access-pt4dk:{mountpoint:/var/lib/kubelet/pods/7e4dc8ce-2802-4864-aed5-affeefe41d6b/volumes/kubernetes.io~projected/kube-api-access-pt4dk major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e4dc8ce-2802-4864-aed5-affeefe41d6b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/7e4dc8ce-2802-4864-aed5-affeefe41d6b/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85cbfa4a-71cd-4b2c-8596-a484deeb15a5/volumes/kubernetes.io~projected/kube-api-access-5wn79:{mountpoint:/var/lib/kubelet/pods/85cbfa4a-71cd-4b2c-8596-a484deeb15a5/volumes/kubernetes.io~projected/kube-api-access-5wn79 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9368ef45-b865-4cdf-95a6-5088fc54277e/volumes/kubernetes.io~projected/kube-api-access-z95xd:{mountpoint:/var/lib/kubelet/pods/9368ef45-b865-4cdf-95a6-5088fc54277e/volumes/kubernetes.io~projected/kube-api-access-z95xd major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97af2a01-5609-4154-a892-5de8ae3e2b9a/volumes/kubernetes.io~projected/kube-api-access-vrzfb:{mountpoint:/var/lib/kubelet/pods/97af2a01-5609-4154-a892-5de8ae3e2b9a/volumes/kubernetes.io~projected/kube-api-access-vrzfb major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a074a9ac-624c-4828-9785-c8f2aecb5827/volumes/kubernetes.io~projected/kube-api-access-5kqjw:{mountpoint:/var/lib/kubelet/pods/a074a9ac-624c-4828-9785-c8f2aecb5827/volumes/kubernetes.io~projected/kube-api-access-5kqjw major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a074a9ac-624c-4828-9785-c8f2aecb5827/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a074a9ac-624c-4828-9785-c8f2aecb5827/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a453427b-5025-4653-a930-ce2cf2f41cce/volumes/kubernetes.io~projected/kube-api-access-m4m8s:{mountpoint:/var/lib/kubelet/pods/a453427b-5025-4653-a930-ce2cf2f41cce/volumes/kubernetes.io~projected/kube-api-access-m4m8s major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4b9af78-fe26-40bc-a4df-126ffd0885d2/volumes/kubernetes.io~projected/kube-api-access-mfjxf:{mountpoint:/var/lib/kubelet/pods/a4b9af78-fe26-40bc-a4df-126ffd0885d2/volumes/kubernetes.io~projected/kube-api-access-mfjxf major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3973c00-9abb-42d0-8a2e-6e657af74f2f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/b3973c00-9abb-42d0-8a2e-6e657af74f2f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3973c00-9abb-42d0-8a2e-6e657af74f2f/volumes/kubernetes.io~projected/kube-api-access-xvpxv:{mountpoint:/var/lib/kubelet/pods/b3973c00-9abb-42d0-8a2e-6e657af74f2f/volumes/kubernetes.io~projected/kube-api-access-xvpxv major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3973c00-9abb-42d0-8a2e-6e657af74f2f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/b3973c00-9abb-42d0-8a2e-6e657af74f2f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b44f96b4-5c43-40cc-9c89-63d4ec368e8b/volumes/kubernetes.io~projected/kube-api-access-xm4fn:{mountpoint:/var/lib/kubelet/pods/b44f96b4-5c43-40cc-9c89-63d4ec368e8b/volumes/kubernetes.io~projected/kube-api-access-xm4fn major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4de14e5-995e-4171-bb61-9f70e777688e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b4de14e5-995e-4171-bb61-9f70e777688e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc4a6db8-98e0-4387-8e64-1020978970c3/volumes/kubernetes.io~projected/kube-api-access-cw824:{mountpoint:/var/lib/kubelet/pods/bc4a6db8-98e0-4387-8e64-1020978970c3/volumes/kubernetes.io~projected/kube-api-access-cw824 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc4a6db8-98e0-4387-8e64-1020978970c3/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/bc4a6db8-98e0-4387-8e64-1020978970c3/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7a1975b-4a3b-426c-977c-8f92d14810ac/volumes/kubernetes.io~projected/kube-api-access-tl4xs:{mountpoint:/var/lib/kubelet/pods/c7a1975b-4a3b-426c-977c-8f92d14810ac/volumes/kubernetes.io~projected/kube-api-access-tl4xs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/daf51d1a-a65f-4fad-8a82-966b8bac7ba4/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/daf51d1a-a65f-4fad-8a82-966b8bac7ba4/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e083a23b-76fd-410d-9633-ece3abc240e1/volumes/kubernetes.io~projected/kube-api-access-dm9gp:{mountpoint:/var/lib/kubelet/pods/e083a23b-76fd-410d-9633-ece3abc240e1/volumes/kubernetes.io~projected/kube-api-access-dm9gp major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f20b07b4-cd27-4410-abb2-c0bd50c01269/volumes/kubernetes.io~projected/kube-api-access-9tdtz:{mountpoint:/var/lib/kubelet/pods/f20b07b4-cd27-4410-abb2-c0bd50c01269/volumes/kubernetes.io~projected/kube-api-access-9tdtz major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7a7133f-8372-49a3-8194-15ed2159aa67/volumes/kubernetes.io~projected/kube-api-access-4g4g4:{mountpoint:/var/lib/kubelet/pods/f7a7133f-8372-49a3-8194-15ed2159aa67/volumes/kubernetes.io~projected/kube-api-access-4g4g4 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbbf8db7-5d42-47b6-b03f-dcd4b68f3211/volumes/kubernetes.io~projected/kube-api-access-slfp9:{mountpoint:/var/lib/kubelet/pods/fbbf8db7-5d42-47b6-b03f-dcd4b68f3211/volumes/kubernetes.io~projected/kube-api-access-slfp9 major:0 minor:201 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/1bfafca4f00affb6a7a6c5c899d4ae34541c2a738a705081a194ea06c5ee8037/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d89cc3e99ba03e0782c049e0389d00e186303460d101d8d0b7231d8c8d49573f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/bd1afdd75ea0ff406f4276fb7e806466d222d2b521036948ea7d643c0d8d4ce8/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/97c58fe64b601714ca935733833e51c330d8f231b466736601771b303b2cb03f/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/220f21513f1fdabc329f227956087fc7e9a81dbab70797fb7ca00369ddf6bf02/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/af6edf7c8e15cd39cdb26e5ce843f0263197ee1d9c3d75ee76897b530f12d159/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/64ee6319a5310b8f86b2de1f5ba04f9ed7901f43a7d887d28b7bf3840173744d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a50ab86cb3285807e9fced799ed7ff88fd05e9ecabb1b17bc91898944940ef8d/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/1e64796178724075c42ae76fde5b9c30bbbd5d1f970c3e81c666e4360ddce676/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/1b67fa1dda225b93a243fccb74f27c0934ae1dbbf695d02688304b237747f8ac/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/8023dc4bffaf7dbf7fc574cb0cd11b7e0046c9d92bbdbc93dca5c529e8b6523e/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/dd964d8dbeaa7f04876baafcadef724f2adcf2ea73776e8bc343066ad5b24b11/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/06154201fc4dfa951af514c4874931cac6b09718bf8df4033dc62c5afbdbba42/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b7318f9c6bd05e3df47a310ed89632cbece0cad6c7cef1d01e231876653adb86/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/89f96512299cff2d7a895d4a8a00fc541552bb54a7c3411d4a92f918c8af20a2/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/403973be0a337dcb6fb3f000266c73c4e2147de685b4fc9e1ab08c830d30f368/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/f9beb794482c6a59a9f4a00b7ae32a467d66aec3c5b5a87bfc3fea44f451188c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/8b3274e172fb8e38e7900ce5eb7d8fbe831262102906dce88a2303ed50863912/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/70b714cec978ce59faf618581d612f8f8d4f0f37ed7cd3681816b034646a6b42/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/4a4bdc43bb4f1d82b0e136b7c4e03e2d468120fdac132322aac2e4544def00c3/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/0b47fb6ccf741e8edc487d46299ad6b62b9d9d19ded323d6f90b7f0d321fac2c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/57774db4004782b6afa35c1fe3dbc9cf61a3ed6879cd26e54ee253c609e82816/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/b5d7fb3237c80341add3e64720b6e5dd6d0139651a4bc6bcbee6074377d91531/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/048ef708e84b36865699e7519e612848ebcabc54f47e831cbd26fbf0c6a5e057/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/08ccfbfcf6ce490a5c6e88ab0d81c7600b6d83a02a0fbe9ac4e96940293d36ea/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/8bdab0af7ce359bb27b998f14ec639405f484e472620cc3d6e34ab9f7c1beafc/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/c3ecce998ee36272d4e7785ad6ff3dac4b685db84bde86af205f592a722fee62/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/3bc9af06d1a0758d72068261624f6fce9063668e44fb444660ef4cefdc671af3/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/04e578dc18de76f39906b84a52b463dbcdf4cba410cea80f8e4ab269cdd55610/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/d43fc6ee50bd7165a0490007d1372e9c0f4caaf014947161ee75628df0e936f0/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/2f3026b7615ff80fa79dbd971e228d36a9fba6c068ede2645d2be61c3000085a/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/a6bd2ce1bbc1052edceab8f7a9683b412b1d2f48845c079a2980135b2c3cb6b1/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/2933ea207830149513ffd5e6202d2b217ec50e4d100c8664b6bd99b492810856/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/261191ffc3eba69b04f020d46cf8547fa8dd5604fcb9ac90503d8fc4dac0c5ae/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/2a8229520f10955ffcbf5727469f5c11a45aa0a3a69f86d969a94bc9f7fdb7f4/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/bf33fd357423b8bb47ec291eb82cb78db9bcbd8af187e55ab3da3c44b127fc6f/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/4a306caa4f15aee3c410a81048203e72a64fa23b380074f08ceae5c654222f77/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/746f97f92a1d11d2f188d0b8ef5b9f02192624a5e457661a464986a7aca14ce7/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/0afa283052a5a39e54b56707220b28f17df8689c3fcceb9f3e8c57d05d355bc8/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/68df3182f87c3d272e64bd20806daac49d09a69956c337a634a2498e218e972d/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/83488c897ee28a6fa03b68cd063bfaddc237f186c3dbe72cedc7d32fee624cc7/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/046534defb04611660bd58977665b0a736716e773cb7806aba72585437ffaafc/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/04f5616b6d8ab6d6b954d7e297df7122263eb387f910be498a413796f3178ffd/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/3ab23c44f00cee6af076d383ec86f464730d7dae079b2d24af96a92dd9888fd0/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/7b86cce2cc96a425acece19a8306547fadfa9b86e7f65019241d5080c45d9589/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/d8c3a121c9aac0e2a61b662fdc38d8febd73e2eb373cf1f8c8f1597addaccf5e/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/261a3e5cab81e735b1c7a7fd09ab9741a104d3ec21dc9375bcf7cf4a8217efff/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/2c1bbe620fe820b0921d9d64a0152ed444bfeb3c116b52f706b1504f73d7390e/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/ccda13f8cf41801e94da6a672c4416aacaf288f457b49143ea046cd8ed5456ec/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/10989973cb3974bd889efea186d2682e76a608de349290dee0b4a0c4e5e35b6e/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/a2d0439959ab8d5cd85e830eff8a4955eef198c77a271238b6b493c179ab4a13/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/302e7728ff1887f1bb32d99752e5c818140127405fd5d11fcc3d372f923bece6/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/df09bd24d0ef47d72af9ca90263a97ceb2ce9adc9d294c4e62ecd11e20977254/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/2f3dded352b4e60125b46586f97c1cc074a54b1210931562f46ae1527a779e11/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 02 17:29:41.436704 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.435813 4860 manager.go:217] Machine: {Timestamp:2026-11-02 17:29:41.435342497 +0000 UTC m=+0.056702284 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099994 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22aac5984598ac859f168d3fa2630c SystemUUID:ec22aac5-9845-98ac-859f-168d3fa2630c BootID:2804984d-2b9b-4698-93b1-9653375ab2ab Filesystems:[{Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/704c917e-c41f-47c3-9a1e-02730361fc8a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4b5c29e0-6bfd-4f43-867a-c460dc227fbc/volumes/kubernetes.io~projected/kube-api-access-n8psf DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dfff4b53674ec62275631a314edb707b7e7f3be4efccafa74bd017d5f347212e/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de2b882663fbc211a07a1f8327255abab18faeefc69d5113bffae1f9a25149c8/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19e5c90135094488bf4817a4198f7e14939531ab670190f348241b436254c4f1/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/97af2a01-5609-4154-a892-5de8ae3e2b9a/volumes/kubernetes.io~projected/kube-api-access-vrzfb DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b3973c00-9abb-42d0-8a2e-6e657af74f2f/volumes/kubernetes.io~projected/kube-api-access-xvpxv DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afadebb9c02b6aa05450a3103a4757f749c4382bb8afebb0b9f81979d07c49f7/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7484d61b-5158-4729-b835-c0b372d8105b/volumes/kubernetes.io~projected/kube-api-access-2b4dt DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1bdf615d401fa21d9ac512784085320be2cb538e86cc41992d168f339019b37e/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd8bb8092f4aed3a7e68285f1a30d1bcaae6ae0c5568d26f8f9bb3a5643cdd85/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f82b9b75a1d021de115fd0758e953cb5e873d00a38ed422f1c672362623e0b0/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5db7566245c6eb0c1454d809ae7c7ffe49dd9a6b0aeb3b2b66136f88fb214087/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7484d61b-5158-4729-b835-c0b372d8105b/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/42f9e736-411e-46a2-89d0-2a3870e2f95a/volumes/kubernetes.io~projected/kube-api-access-8jf4x DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4072df9c777140fe13d8e46d3c3c46ce4d4842f2cdb8f0e9ad3db97af69c50a3/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35669b5e-5c97-45ef-8868-4cff3fdfc07a/volumes/kubernetes.io~projected/kube-api-access-lrvfr DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9368ef45-b865-4cdf-95a6-5088fc54277e/volumes/kubernetes.io~projected/kube-api-access-z95xd DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d83bf78-e7b7-48bb-98be-8196e92fd320/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7484d61b-5158-4729-b835-c0b372d8105b/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f9e8821-40a8-4012-b66a-813e80d856f2/volumes/kubernetes.io~projected/kube-api-access-spg94 DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/732370e3f644cd5f750ef139deb4731284af5b0051284347014685a23ed3b579/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38e5c802-99d4-4234-9e64-dee8e19b055a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc4a6db8-98e0-4387-8e64-1020978970c3/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3d83bf78-e7b7-48bb-98be-8196e92fd320/volumes/kubernetes.io~projected/kube-api-access-mm8pr DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7a7133f-8372-49a3-8194-15ed2159aa67/volumes/kubernetes.io~projected/kube-api-access-4g4g4 DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fbbf8db7-5d42-47b6-b03f-dcd4b68f3211/volumes/kubernetes.io~projected/kube-api-access-slfp9 DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69bf746fbe676b165b8dc071761c74a42b201c4e3db51d6f2afcd1b14ea86d4e/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7484d61b-5158-4729-b835-c0b372d8105b/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b127ce3c4bb20da36b450f6d58d6b07b125f3172440deda51dbb06f211cd4c5b/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c35565f0de21fb64553bae63d86fa9635c3e543017dd84bf2ba17486215cda7a/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27ea270b-6e55-4652-a1bf-1431a21c3d55/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6d4af786dd97bc313a2a112a07af8b16a79f2f3d4ef347307b54d3916c868c6/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/38e5c802-99d4-4234-9e64-dee8e19b055a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e083a23b-76fd-410d-9633-ece3abc240e1/volumes/kubernetes.io~projected/kube-api-access-dm9gp DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3d83bf78-e7b7-48bb-98be-8196e92fd320/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7e4dc8ce-2802-4864-aed5-affeefe41d6b/volumes/kubernetes.io~projected/kube-api-access-pt4dk DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c38e5e87ea0feaac1a2bf835454b3c0968e9442f8fba1d8225f9b3565583717/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5dd2759fbe8d71f6c45700e5a83443fcb3baac4c517e585e0e7245c0e956460/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27ea270b-6e55-4652-a1bf-1431a21c3d55/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a074a9ac-624c-4828-9785-c8f2aecb5827/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f20b07b4-cd27-4410-abb2-c0bd50c01269/volumes/kubernetes.io~projected/kube-api-access-9tdtz DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a074a9ac-624c-4828-9785-c8f2aecb5827/volumes/kubernetes.io~projected/kube-api-access-5kqjw DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5e336b15af15f9570964afe8da5675163e1868f984595421ce2c9f2a07cb7e2/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39f2630e80b307bd7a346a66e784461be5ae36dad8ab9e62145a42b163f9c45d/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61880416d13b3c700b791ab52cfdc2ebbc6aa31c487736166dc2bbe25b665e32/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85cbfa4a-71cd-4b2c-8596-a484deeb15a5/volumes/kubernetes.io~projected/kube-api-access-5wn79 DeviceMajor:0 DeviceMinor:310 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b3973c00-9abb-42d0-8a2e-6e657af74f2f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/7e4dc8ce-2802-4864-aed5-affeefe41d6b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/38e5c802-99d4-4234-9e64-dee8e19b055a/volumes/kubernetes.io~projected/kube-api-access-jphfq DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/704c917e-c41f-47c3-9a1e-02730361fc8a/volumes/kubernetes.io~projected/kube-api-access-jd9gl DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b44f96b4-5c43-40cc-9c89-63d4ec368e8b/volumes/kubernetes.io~projected/kube-api-access-xm4fn DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afb720b260f19c89ec5d6fd564551c31801043e60c91fe6cd422f21f3a143d88/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/daf51d1a-a65f-4fad-8a82-966b8bac7ba4/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46d2456f-62b2-4aa5-b172-1c28ab7615ac/volumes/kubernetes.io~projected/kube-api-access-wqrgs DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a453427b-5025-4653-a930-ce2cf2f41cce/volumes/kubernetes.io~projected/kube-api-access-m4m8s DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/533e0dcf6ba7212004f611f9b2995fc59b1eb2eff0cb758582001a69e7cddbad/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27ea270b-6e55-4652-a1bf-1431a21c3d55/volumes/kubernetes.io~projected/kube-api-access-rpdh7 DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7a1975b-4a3b-426c-977c-8f92d14810ac/volumes/kubernetes.io~projected/kube-api-access-tl4xs DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0eacac81-35d0-4ab1-bea4-25aaefa74c8b/volumes/kubernetes.io~projected/kube-api-access-njp5v DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74924f8e0f80598f0ae2d2226505477ff26bbd3e35ef383ffd97234d57604a1e/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b4de14e5-995e-4171-bb61-9f70e777688e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/bc4a6db8-98e0-4387-8e64-1020978970c3/volumes/kubernetes.io~projected/kube-api-access-cw824 DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b3973c00-9abb-42d0-8a2e-6e657af74f2f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/27ea270b-6e55-4652-a1bf-1431a21c3d55/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5f9e8821-40a8-4012-b66a-813e80d856f2/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4b533156-8ec1-4994-9bf6-75e4c41c3c6a/volumes/kubernetes.io~projected/kube-api-access-zdlkh DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a4b9af78-fe26-40bc-a4df-126ffd0885d2/volumes/kubernetes.io~projected/kube-api-access-mfjxf DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:19e5c9013509448 MacAddress:02:b7:ac:4a:ab:ff Speed:10000 Mtu:8901} {Name:1bdf615d401fa21 MacAddress:be:90:1d:98:bc:cc Speed:10000 Mtu:8901} {Name:39f2630e80b307b MacAddress:ea:bc:d4:a6:62:9f Speed:10000 Mtu:8901} {Name:4072df9c777140f MacAddress:2e:77:73:6e:fc:66 Speed:10000 Mtu:8901} {Name:732370e3f644cd5 MacAddress:da:39:7d:44:9f:fc Speed:10000 Mtu:8901} {Name:9c38e5e87ea0fea MacAddress:96:29:0d:dc:b6:9f Speed:10000 Mtu:8901} {Name:b127ce3c4bb20da MacAddress:22:39:a0:3c:f4:30 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d8:2a:eb:cb Speed:0 Mtu:9001} {Name:br-int MacAddress:ae:16:3d:04:d8:7e Speed:0 Mtu:8901} {Name:c35565f0de21fb6 MacAddress:e2:7f:c8:4e:c0:3a Speed:10000 Mtu:8901} {Name:cd8bb8092f4aed3 MacAddress:16:6d:c4:28:d4:e7 Speed:10000 Mtu:8901} {Name:de2b882663fbc21 MacAddress:26:34:6f:66:03:22 Speed:10000 Mtu:8901} {Name:dfff4b53674ec62 MacAddress:16:b3:be:9a:7a:96 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d8:2a:eb:cb Speed:0 Mtu:9001} {Name:f5e336b15af15f9 MacAddress:12:2d:98:5d:2c:19 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:a6:6f:0c:53:d0:00 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:56:d2:fa:56:ac:72 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 17:29:41.436704 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.436169 4860 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 17:29:41.436704 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.436216 4860 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 17:29:41.436704 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.436497 4860 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 17:29:41.436704 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.436517 4860 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-153.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 17:29:41.436704 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.436641 4860 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 17:29:41.436704 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.436647 4860 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 17:29:41.436704 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.436657 4860 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:29:41.436704 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.436677 4860 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:29:41.436704 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.436694 4860 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 17:29:41.437169 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.436872 4860 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 17:29:41.437169 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.436981 4860 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 17:29:41.437169 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.437022 4860 kubelet.go:493] "Attempting to sync node with API server" Nov 02 17:29:41.437169 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.437036 4860 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 17:29:41.437169 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.437052 4860 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 17:29:41.437169 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.437065 4860 kubelet.go:405] "Adding apiserver pod source" Nov 02 17:29:41.437169 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.437075 4860 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 17:29:41.437985 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.437970 4860 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 17:29:41.437985 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.437984 4860 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:29:41.438407 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.438386 4860 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 17:29:41.440546 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.440527 4860 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 17:29:41.440546 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.440542 4860 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:29:41.440706 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.440662 4860 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 17:29:41.440950 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.440933 4860 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 17:29:41.441019 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.440974 4860 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 17:29:41.441105 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.441094 4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 17:29:41.441159 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.441112 4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 17:29:41.441159 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.441124 4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 17:29:41.441159 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.441134 4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 17:29:41.441159 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.441144 4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 17:29:41.441159 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.441154 4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 17:29:41.441389 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.441165 4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 17:29:41.441389 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.441174 4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 17:29:41.441389 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.441185 4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 17:29:41.441389 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.441194 4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 17:29:41.441389 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.441214 4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 17:29:41.441389 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.441229 4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 17:29:41.441389 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.441263 4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 17:29:41.441389 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.441273 4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 17:29:41.444706 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.444689 4860 server.go:1263] "Started kubelet" Nov 02 17:29:41.444793 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.444747 4860 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 17:29:41.451442 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.444879 4860 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 17:29:41.451442 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.445252 4860 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 17:29:41.451442 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.445548 4860 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 17:29:41.451442 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.448119 4860 server.go:317] "Adding debug handlers to kubelet server" Nov 02 17:29:41.451442 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.449716 4860 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 17:29:41.451442 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.449739 4860 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 17:29:41.451442 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.449781 4860 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 17:29:41.451442 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.449808 4860 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 17:29:41.451442 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:41.449826 4860 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-153.ec2.internal\" not found" Nov 02 17:29:41.451442 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.449939 4860 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:24:03 +0000 UTC" deadline="2028-08-13 10:54:25.554017474 +0000 UTC" Nov 02 17:29:41.451442 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.449959 4860 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 17:29:41.451442 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.449965 4860 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15593h24m44.104055543s" Nov 02 17:29:41.445719 ip-10-0-136-153 systemd[1]: Started Kubernetes Kubelet. Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.451652 4860 factory.go:55] Registering systemd factory Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.451663 4860 factory.go:223] Registration of the systemd container factory successfully Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452713 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ea270b-6e55-4652-a1bf-1431a21c3d55" volumeName="kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-certificates" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452741 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7484d61b-5158-4729-b835-c0b372d8105b" volumeName="kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-trusted-ca" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452750 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7484d61b-5158-4729-b835-c0b372d8105b" volumeName="kubernetes.io/empty-dir/7484d61b-5158-4729-b835-c0b372d8105b-ca-trust-extracted" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452760 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a074a9ac-624c-4828-9785-c8f2aecb5827" volumeName="kubernetes.io/secret/a074a9ac-624c-4828-9785-c8f2aecb5827-serving-cert" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452767 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daf51d1a-a65f-4fad-8a82-966b8bac7ba4" volumeName="kubernetes.io/configmap/daf51d1a-a65f-4fad-8a82-966b8bac7ba4-konnectivity-ca" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452776 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f20b07b4-cd27-4410-abb2-c0bd50c01269" volumeName="kubernetes.io/configmap/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-config" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452788 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7a7133f-8372-49a3-8194-15ed2159aa67" volumeName="kubernetes.io/projected/f7a7133f-8372-49a3-8194-15ed2159aa67-kube-api-access-4g4g4" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452796 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ea270b-6e55-4652-a1bf-1431a21c3d55" volumeName="kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-kube-api-access-rpdh7" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452805 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42f9e736-411e-46a2-89d0-2a3870e2f95a" volumeName="kubernetes.io/configmap/42f9e736-411e-46a2-89d0-2a3870e2f95a-telemetry-config" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452813 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a074a9ac-624c-4828-9785-c8f2aecb5827" volumeName="kubernetes.io/configmap/a074a9ac-624c-4828-9785-c8f2aecb5827-config" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452820 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3973c00-9abb-42d0-8a2e-6e657af74f2f" volumeName="kubernetes.io/configmap/b3973c00-9abb-42d0-8a2e-6e657af74f2f-env-overrides" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452828 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a453427b-5025-4653-a930-ce2cf2f41cce" volumeName="kubernetes.io/configmap/a453427b-5025-4653-a930-ce2cf2f41cce-config-volume" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452836 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38e5c802-99d4-4234-9e64-dee8e19b055a" volumeName="kubernetes.io/empty-dir/38e5c802-99d4-4234-9e64-dee8e19b055a-tmp" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452843 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72445bf5-a7a3-49dc-9b97-5589ae901032" volumeName="kubernetes.io/configmap/72445bf5-a7a3-49dc-9b97-5589ae901032-nginx-conf" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452882 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7484d61b-5158-4729-b835-c0b372d8105b" volumeName="kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-installation-pull-secrets" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452895 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85cbfa4a-71cd-4b2c-8596-a484deeb15a5" volumeName="kubernetes.io/projected/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-api-access-5wn79" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452906 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a453427b-5025-4653-a930-ce2cf2f41cce" volumeName="kubernetes.io/empty-dir/a453427b-5025-4653-a930-ce2cf2f41cce-tmp-dir" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452919 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a453427b-5025-4653-a930-ce2cf2f41cce" volumeName="kubernetes.io/projected/a453427b-5025-4653-a930-ce2cf2f41cce-kube-api-access-m4m8s" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452930 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b44f96b4-5c43-40cc-9c89-63d4ec368e8b" volumeName="kubernetes.io/configmap/b44f96b4-5c43-40cc-9c89-63d4ec368e8b-serviceca" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452942 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ea270b-6e55-4652-a1bf-1431a21c3d55" volumeName="kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-bound-sa-token" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452962 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38e5c802-99d4-4234-9e64-dee8e19b055a" volumeName="kubernetes.io/projected/38e5c802-99d4-4234-9e64-dee8e19b055a-kube-api-access-jphfq" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452975 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9e8821-40a8-4012-b66a-813e80d856f2" volumeName="kubernetes.io/configmap/5f9e8821-40a8-4012-b66a-813e80d856f2-trusted-ca" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452988 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="704c917e-c41f-47c3-9a1e-02730361fc8a" volumeName="kubernetes.io/secret/704c917e-c41f-47c3-9a1e-02730361fc8a-cert" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.452999 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3973c00-9abb-42d0-8a2e-6e657af74f2f" volumeName="kubernetes.io/projected/b3973c00-9abb-42d0-8a2e-6e657af74f2f-kube-api-access-xvpxv" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453006 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4de14e5-995e-4171-bb61-9f70e777688e" volumeName="kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453014 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc4a6db8-98e0-4387-8e64-1020978970c3" volumeName="kubernetes.io/projected/bc4a6db8-98e0-4387-8e64-1020978970c3-kube-api-access-cw824" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453022 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35669b5e-5c97-45ef-8868-4cff3fdfc07a" volumeName="kubernetes.io/configmap/35669b5e-5c97-45ef-8868-4cff3fdfc07a-cni-binary-copy" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453029 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="704c917e-c41f-47c3-9a1e-02730361fc8a" volumeName="kubernetes.io/projected/704c917e-c41f-47c3-9a1e-02730361fc8a-kube-api-access-jd9gl" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453041 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4b9af78-fe26-40bc-a4df-126ffd0885d2" volumeName="kubernetes.io/configmap/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-daemon-config" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453053 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daf51d1a-a65f-4fad-8a82-966b8bac7ba4" volumeName="kubernetes.io/secret/daf51d1a-a65f-4fad-8a82-966b8bac7ba4-agent-certs" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453061 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ea270b-6e55-4652-a1bf-1431a21c3d55" volumeName="kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-installation-pull-secrets" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453070 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38e5c802-99d4-4234-9e64-dee8e19b055a" volumeName="kubernetes.io/empty-dir/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-tuned" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453077 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7484d61b-5158-4729-b835-c0b372d8105b" volumeName="kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-image-registry-private-configuration" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453085 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35669b5e-5c97-45ef-8868-4cff3fdfc07a" volumeName="kubernetes.io/projected/35669b5e-5c97-45ef-8868-4cff3fdfc07a-kube-api-access-lrvfr" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453095 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97af2a01-5609-4154-a892-5de8ae3e2b9a" volumeName="kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453107 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e4dc8ce-2802-4864-aed5-affeefe41d6b" volumeName="kubernetes.io/configmap/7e4dc8ce-2802-4864-aed5-affeefe41d6b-trusted-ca-bundle" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453119 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ea270b-6e55-4652-a1bf-1431a21c3d55" volumeName="kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-image-registry-private-configuration" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453131 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d83bf78-e7b7-48bb-98be-8196e92fd320" volumeName="kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-default-certificate" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453142 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9e8821-40a8-4012-b66a-813e80d856f2" volumeName="kubernetes.io/projected/5f9e8821-40a8-4012-b66a-813e80d856f2-kube-api-access-spg94" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453155 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7484d61b-5158-4729-b835-c0b372d8105b" volumeName="kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-registry-certificates" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453166 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7484d61b-5158-4729-b835-c0b372d8105b" volumeName="kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-bound-sa-token" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453178 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e4dc8ce-2802-4864-aed5-affeefe41d6b" volumeName="kubernetes.io/empty-dir/7e4dc8ce-2802-4864-aed5-affeefe41d6b-tmp" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453190 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e4dc8ce-2802-4864-aed5-affeefe41d6b" volumeName="kubernetes.io/projected/7e4dc8ce-2802-4864-aed5-affeefe41d6b-kube-api-access-pt4dk" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453201 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3973c00-9abb-42d0-8a2e-6e657af74f2f" volumeName="kubernetes.io/secret/b3973c00-9abb-42d0-8a2e-6e657af74f2f-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453214 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35669b5e-5c97-45ef-8868-4cff3fdfc07a" volumeName="kubernetes.io/configmap/35669b5e-5c97-45ef-8868-4cff3fdfc07a-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453226 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e4dc8ce-2802-4864-aed5-affeefe41d6b" volumeName="kubernetes.io/empty-dir/7e4dc8ce-2802-4864-aed5-affeefe41d6b-snapshots" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453237 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f20b07b4-cd27-4410-abb2-c0bd50c01269" volumeName="kubernetes.io/projected/f20b07b4-cd27-4410-abb2-c0bd50c01269-kube-api-access-9tdtz" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453248 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ea270b-6e55-4652-a1bf-1431a21c3d55" volumeName="kubernetes.io/empty-dir/27ea270b-6e55-4652-a1bf-1431a21c3d55-ca-trust-extracted" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453262 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46d2456f-62b2-4aa5-b172-1c28ab7615ac" volumeName="kubernetes.io/projected/46d2456f-62b2-4aa5-b172-1c28ab7615ac-kube-api-access-wqrgs" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453272 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b5c29e0-6bfd-4f43-867a-c460dc227fbc" volumeName="kubernetes.io/projected/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-kube-api-access-n8psf" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453283 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7484d61b-5158-4729-b835-c0b372d8105b" volumeName="kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-kube-api-access-2b4dt" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453295 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0eacac81-35d0-4ab1-bea4-25aaefa74c8b" volumeName="kubernetes.io/projected/0eacac81-35d0-4ab1-bea4-25aaefa74c8b-kube-api-access-njp5v" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453307 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42f9e736-411e-46a2-89d0-2a3870e2f95a" volumeName="kubernetes.io/projected/42f9e736-411e-46a2-89d0-2a3870e2f95a-kube-api-access-8jf4x" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453318 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b533156-8ec1-4994-9bf6-75e4c41c3c6a" volumeName="kubernetes.io/projected/4b533156-8ec1-4994-9bf6-75e4c41c3c6a-kube-api-access-zdlkh" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453330 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9e8821-40a8-4012-b66a-813e80d856f2" volumeName="kubernetes.io/configmap/5f9e8821-40a8-4012-b66a-813e80d856f2-config" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453341 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9368ef45-b865-4cdf-95a6-5088fc54277e" volumeName="kubernetes.io/empty-dir/9368ef45-b865-4cdf-95a6-5088fc54277e-tmp-dir" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453364 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9368ef45-b865-4cdf-95a6-5088fc54277e" volumeName="kubernetes.io/projected/9368ef45-b865-4cdf-95a6-5088fc54277e-kube-api-access-z95xd" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453378 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a074a9ac-624c-4828-9785-c8f2aecb5827" volumeName="kubernetes.io/projected/a074a9ac-624c-4828-9785-c8f2aecb5827-kube-api-access-5kqjw" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453391 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4b9af78-fe26-40bc-a4df-126ffd0885d2" volumeName="kubernetes.io/projected/a4b9af78-fe26-40bc-a4df-126ffd0885d2-kube-api-access-mfjxf" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453403 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85cbfa4a-71cd-4b2c-8596-a484deeb15a5" volumeName="kubernetes.io/empty-dir/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-data-volume" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453415 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3973c00-9abb-42d0-8a2e-6e657af74f2f" volumeName="kubernetes.io/configmap/b3973c00-9abb-42d0-8a2e-6e657af74f2f-ovnkube-config" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453427 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc4a6db8-98e0-4387-8e64-1020978970c3" volumeName="kubernetes.io/secret/bc4a6db8-98e0-4387-8e64-1020978970c3-serving-cert" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453440 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7a1975b-4a3b-426c-977c-8f92d14810ac" volumeName="kubernetes.io/projected/c7a1975b-4a3b-426c-977c-8f92d14810ac-kube-api-access-tl4xs" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453454 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d83bf78-e7b7-48bb-98be-8196e92fd320" volumeName="kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-stats-auth" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453469 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9e8821-40a8-4012-b66a-813e80d856f2" volumeName="kubernetes.io/secret/5f9e8821-40a8-4012-b66a-813e80d856f2-serving-cert" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453482 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e4dc8ce-2802-4864-aed5-affeefe41d6b" volumeName="kubernetes.io/configmap/7e4dc8ce-2802-4864-aed5-affeefe41d6b-service-ca-bundle" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453493 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e4dc8ce-2802-4864-aed5-affeefe41d6b" volumeName="kubernetes.io/secret/7e4dc8ce-2802-4864-aed5-affeefe41d6b-serving-cert" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453506 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3973c00-9abb-42d0-8a2e-6e657af74f2f" volumeName="kubernetes.io/configmap/b3973c00-9abb-42d0-8a2e-6e657af74f2f-ovnkube-script-lib" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453519 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc4a6db8-98e0-4387-8e64-1020978970c3" volumeName="kubernetes.io/configmap/bc4a6db8-98e0-4387-8e64-1020978970c3-config" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453532 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ea270b-6e55-4652-a1bf-1431a21c3d55" volumeName="kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-trusted-ca" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453544 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4b9af78-fe26-40bc-a4df-126ffd0885d2" volumeName="kubernetes.io/configmap/a4b9af78-fe26-40bc-a4df-126ffd0885d2-cni-binary-copy" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453557 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b44f96b4-5c43-40cc-9c89-63d4ec368e8b" volumeName="kubernetes.io/projected/b44f96b4-5c43-40cc-9c89-63d4ec368e8b-kube-api-access-xm4fn" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453570 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e083a23b-76fd-410d-9633-ece3abc240e1" volumeName="kubernetes.io/projected/e083a23b-76fd-410d-9633-ece3abc240e1-kube-api-access-dm9gp" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453583 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35669b5e-5c97-45ef-8868-4cff3fdfc07a" volumeName="kubernetes.io/configmap/35669b5e-5c97-45ef-8868-4cff3fdfc07a-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453597 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d83bf78-e7b7-48bb-98be-8196e92fd320" volumeName="kubernetes.io/projected/3d83bf78-e7b7-48bb-98be-8196e92fd320-kube-api-access-mm8pr" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453608 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b533156-8ec1-4994-9bf6-75e4c41c3c6a" volumeName="kubernetes.io/configmap/4b533156-8ec1-4994-9bf6-75e4c41c3c6a-iptables-alerter-script" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453620 4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbbf8db7-5d42-47b6-b03f-dcd4b68f3211" volumeName="kubernetes.io/projected/fbbf8db7-5d42-47b6-b03f-dcd4b68f3211-kube-api-access-slfp9" seLinuxMountContext="" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453632 4860 reconstruct.go:97] "Volume reconstruction finished" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453641 4860 reconciler.go:29] "Reconciler: start to sync state" Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453826 4860 factory.go:153] Registering CRI-O factory Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453842 4860 factory.go:223] Registration of the crio container factory successfully Nov 02 17:29:41.453888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453926 4860 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 17:29:41.458295 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453956 4860 factory.go:103] Registering Raw factory Nov 02 17:29:41.458295 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.453978 4860 manager.go:1199] Started watching for new ooms in manager Nov 02 17:29:41.458295 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.454452 4860 manager.go:319] Starting recovery of all containers Nov 02 17:29:41.472393 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.472369 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/1.log" Nov 02 17:29:41.472516 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.472441 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/1.log" Nov 02 17:29:41.488973 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.488930 4860 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 17:29:41.490613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.490581 4860 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 17:29:41.490613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.490610 4860 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 17:29:41.490731 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.490638 4860 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 17:29:41.490731 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:41.490684 4860 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 17:29:41.493564 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.493536 4860 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:29:41.500075 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.500057 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/1.log" Nov 02 17:29:41.500169 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.500087 4860 generic.go:356] "Generic (PLEG): container finished" podID="5f9e8821-40a8-4012-b66a-813e80d856f2" containerID="3d9ba56bcbd5f834961a29159d8acb53f61960759a915226ffb42f05b8a1ea33" exitCode=255 Nov 02 17:29:41.506404 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.506384 4860 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:29:41.506589 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.506565 4860 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 17:29:41.519472 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.519449 4860 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:29:41.519821 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.519801 4860 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:29:41.522300 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.522278 4860 generic.go:356] "Generic (PLEG): container finished" podID="35669b5e-5c97-45ef-8868-4cff3fdfc07a" containerID="979dd8973ce0cdc9593a794853beec352f3a2ced6640fb69e611a79c49708ed2" exitCode=0 Nov 02 17:29:41.522300 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.522296 4860 generic.go:356] "Generic (PLEG): container finished" podID="35669b5e-5c97-45ef-8868-4cff3fdfc07a" containerID="83fb1ea4639c3dd44d748513affde2f4993ef1eea1f3a91afa1558dda8a18e13" exitCode=0 Nov 02 17:29:41.522300 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.522302 4860 generic.go:356] "Generic (PLEG): container finished" podID="35669b5e-5c97-45ef-8868-4cff3fdfc07a" containerID="cc386ebdd2c93e26c30046b8b86651bc02e26f92708ff389c7b08fde0ee81032" exitCode=0 Nov 02 17:29:41.522428 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.522310 4860 generic.go:356] "Generic (PLEG): container finished" podID="35669b5e-5c97-45ef-8868-4cff3fdfc07a" containerID="aea72000649cfc0d5cdc04e21f53555cc9cc4f95d2af63e8a2ba620f3b91fe5d" exitCode=0 Nov 02 17:29:41.522428 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.522316 4860 generic.go:356] "Generic (PLEG): container finished" podID="35669b5e-5c97-45ef-8868-4cff3fdfc07a" containerID="c29d18cba08ab53e65ed59371193eb27c1ad5ab972a6c38218c88e7131872cb7" exitCode=0 Nov 02 17:29:41.522428 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.522323 4860 generic.go:356] "Generic (PLEG): container finished" podID="35669b5e-5c97-45ef-8868-4cff3fdfc07a" containerID="21b80520cefbee2633d6adb4be4d11f4252a293fe5a9650107a5cf0af481fe52" exitCode=0 Nov 02 17:29:41.543594 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.543567 4860 generic.go:356] "Generic (PLEG): container finished" podID="6d9161e9f087fe7b62acf0194feffaf9" containerID="cffaa3e6baf86786d95ebb88993c6c7957078f225f3f26e540b998ad31595943" exitCode=0 Nov 02 17:29:41.550987 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.550962 4860 manager.go:324] Recovery completed Nov 02 17:29:41.569388 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.569330 4860 cpu_manager.go:226] "Starting" policy="none" Nov 02 17:29:41.569388 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.569344 4860 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 17:29:41.569388 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.569361 4860 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 17:29:41.569579 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.569467 4860 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 17:29:41.569579 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.569478 4860 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 17:29:41.569579 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.569494 4860 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 17:29:41.569579 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.569499 4860 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 17:29:41.569579 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.569504 4860 policy_none.go:50] "Start" Nov 02 17:29:41.569579 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.569513 4860 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 17:29:41.569579 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.569521 4860 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 17:29:41.587123 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.587100 4860 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 17:29:41.587123 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.587117 4860 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 17:29:41.587123 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.587124 4860 policy_none.go:44] "Start" Nov 02 17:29:41.590763 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:41.590740 4860 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 17:29:41.593843 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.593817 4860 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 17:29:41.598234 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.598217 4860 manager.go:341] "Starting Device Plugin manager" Nov 02 17:29:41.598719 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:41.598246 4860 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 17:29:41.598719 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.598720 4860 server.go:87] "Starting device plugin registration server" Nov 02 17:29:41.599011 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.598999 4860 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 17:29:41.599064 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.599011 4860 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 17:29:41.599135 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.599119 4860 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 17:29:41.599407 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.599213 4860 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 17:29:41.599407 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.599253 4860 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 17:29:41.599407 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.599260 4860 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 17:29:41.599407 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.599303 4860 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:29:41.599242519Z","UUID":"c61b0129-bad9-43ea-8649-cfa29d4ca2c8","Handler":null,"Name":"","Endpoint":""} Nov 02 17:29:41.600648 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.600633 4860 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 17:29:41.600727 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.600658 4860 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 17:29:41.600779 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:41.600751 4860 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 17:29:41.699973 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.699945 4860 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:29:41.701307 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.701289 4860 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasSufficientMemory" Nov 02 17:29:41.701415 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.701322 4860 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 17:29:41.701415 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.701338 4860 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-153.ec2.internal" event="NodeHasSufficientPID" Nov 02 17:29:41.701415 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.701393 4860 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-153.ec2.internal" Nov 02 17:29:41.743523 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.743497 4860 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-153.ec2.internal" Nov 02 17:29:41.743636 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.743573 4860 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-153.ec2.internal" Nov 02 17:29:41.790899 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.790824 4860 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal"] Nov 02 17:29:41.791517 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.791482 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" event={"ID":"6d9161e9f087fe7b62acf0194feffaf9","Type":"ContainerStarted","Data":"3d0e91257f8e78ff700016a6613775fb304037d0fbfb1e9add6216e0889104f1"} Nov 02 17:29:41.791571 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.791519 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" event={"ID":"6d9161e9f087fe7b62acf0194feffaf9","Type":"ContainerDied","Data":"cffaa3e6baf86786d95ebb88993c6c7957078f225f3f26e540b998ad31595943"} Nov 02 17:29:41.791571 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.791533 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" event={"ID":"6d9161e9f087fe7b62acf0194feffaf9","Type":"ContainerStarted","Data":"533e0dcf6ba7212004f611f9b2995fc59b1eb2eff0cb758582001a69e7cddbad"} Nov 02 17:29:41.791571 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.791545 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal" event={"ID":"e3f259cc4f9c47e64336e525d3250602","Type":"ContainerStarted","Data":"b5d90f2023d29bdcbffcaee7e9552c5099bcd555864df4d4d2c64d4abaea5c06"} Nov 02 17:29:41.791571 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.791556 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal" event={"ID":"e3f259cc4f9c47e64336e525d3250602","Type":"ContainerStarted","Data":"f6d4af786dd97bc313a2a112a07af8b16a79f2f3d4ef347307b54d3916c868c6"} Nov 02 17:29:41.791571 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.791563 4860 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal" Nov 02 17:29:41.791736 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.791601 4860 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" Nov 02 17:29:41.818316 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.818293 4860 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:29:41.818400 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:41.818342 4860 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-153.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal" Nov 02 17:29:41.820918 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.820877 4860 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:29:41.820918 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:41.820913 4860 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" Nov 02 17:29:41.908292 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.908262 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6d9161e9f087fe7b62acf0194feffaf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal\" (UID: \"6d9161e9f087fe7b62acf0194feffaf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" Nov 02 17:29:41.908292 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.908294 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d9161e9f087fe7b62acf0194feffaf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal\" (UID: \"6d9161e9f087fe7b62acf0194feffaf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" Nov 02 17:29:41.908444 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:41.908317 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3f259cc4f9c47e64336e525d3250602-config\") pod \"kube-apiserver-proxy-ip-10-0-136-153.ec2.internal\" (UID: \"e3f259cc4f9c47e64336e525d3250602\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal" Nov 02 17:29:42.008924 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.008889 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6d9161e9f087fe7b62acf0194feffaf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal\" (UID: \"6d9161e9f087fe7b62acf0194feffaf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" Nov 02 17:29:42.009048 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.008927 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d9161e9f087fe7b62acf0194feffaf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal\" (UID: \"6d9161e9f087fe7b62acf0194feffaf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" Nov 02 17:29:42.009048 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.008958 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3f259cc4f9c47e64336e525d3250602-config\") pod \"kube-apiserver-proxy-ip-10-0-136-153.ec2.internal\" (UID: \"e3f259cc4f9c47e64336e525d3250602\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal" Nov 02 17:29:42.009048 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.009000 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6d9161e9f087fe7b62acf0194feffaf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal\" (UID: \"6d9161e9f087fe7b62acf0194feffaf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" Nov 02 17:29:42.009048 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.009019 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d9161e9f087fe7b62acf0194feffaf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal\" (UID: \"6d9161e9f087fe7b62acf0194feffaf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" Nov 02 17:29:42.009048 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.009006 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3f259cc4f9c47e64336e525d3250602-config\") pod \"kube-apiserver-proxy-ip-10-0-136-153.ec2.internal\" (UID: \"e3f259cc4f9c47e64336e525d3250602\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal" Nov 02 17:29:42.437814 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.437778 4860 apiserver.go:52] "Watching apiserver" Nov 02 17:29:42.496664 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.496629 4860 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 17:29:42.497278 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.497258 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tszgp","kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal","openshift-multus/network-metrics-daemon-g6b2c","openshift-ingress-canary/ingress-canary-4qfkf","openshift-insights/insights-runtime-extractor-fbrw7","openshift-network-console/networking-console-plugin-86c478bb88-7scpf","openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn","openshift-network-operator/iptables-alerter-bmw6p","openshift-ingress/router-default-79c77b5cd5-c6svb","kube-system/kas-connection-checker-685895c7c9-nxnkz","openshift-dns/node-resolver-s6s58","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal","openshift-dns/dns-default-w7zsl","kube-system/konnectivity-agent-vs4r4","openshift-network-diagnostics/network-check-target-qjx25","openshift-image-registry/image-registry-78749f569b-zzhxm","kube-system/global-pull-secret-syncer-w8vvj","openshift-console-operator/console-operator-7cd56b8dbc-xqzqj","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn","kube-system/kas-connection-checker-685895c7c9-hxpgv","openshift-network-diagnostics/network-check-source-6f78d59f99-5fvrw","openshift-ovn-kubernetes/ovnkube-node-k58q5","openshift-multus/multus-additional-cni-plugins-p6d66","openshift-image-registry/image-registry-6f8ffbdd9-xqf4t","openshift-image-registry/node-ca-jxsgc","openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk","kube-system/kas-connection-checker-685895c7c9-7jhzf","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dmr89","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr","openshift-cluster-node-tuning-operator/tuned-ptdb2","openshift-multus/multus-h6dft"] Nov 02 17:29:42.498714 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.498695 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:42.499697 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.499677 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:42.500068 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.500046 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:42.500651 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.500635 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:42.501791 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.501775 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:42.508696 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.508672 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:42.510245 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.510231 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:42.511398 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.511383 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:42.512429 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.512412 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:42.517965 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.517948 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 17:29:42.518012 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.517964 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 17:29:42.518012 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.517950 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 17:29:42.518012 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.518006 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-c5qgh\"" Nov 02 17:29:42.518266 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.518253 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 17:29:42.518312 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.518285 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 17:29:42.518609 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.518598 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 17:29:42.518927 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.518911 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mnv5k\"" Nov 02 17:29:42.519635 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.519621 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 17:29:42.519693 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.519635 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 17:29:42.519693 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.519635 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:42.520735 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.520719 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 17:29:42.551426 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.551400 4860 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" Nov 02 17:29:42.551512 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.551477 4860 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal" Nov 02 17:29:42.552363 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.552348 4860 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 17:29:42.590949 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.590921 4860 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:29:42.591067 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.590979 4860 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-153.ec2.internal" Nov 02 17:29:42.598596 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.598569 4860 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:29:42.598638 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.598616 4860 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-153.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-153.ec2.internal" Nov 02 17:29:42.610917 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.610897 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35669b5e-5c97-45ef-8868-4cff3fdfc07a-cnibin\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:42.610986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.610925 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfjxf\" (UniqueName: \"kubernetes.io/projected/a4b9af78-fe26-40bc-a4df-126ffd0885d2-kube-api-access-mfjxf\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.610986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.610944 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/daf51d1a-a65f-4fad-8a82-966b8bac7ba4-agent-certs\") pod \"konnectivity-agent-vs4r4\" (UID: \"daf51d1a-a65f-4fad-8a82-966b8bac7ba4\") " pod="kube-system/konnectivity-agent-vs4r4" Nov 02 17:29:42.610986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.610961 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-bound-sa-token\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:42.611133 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.610988 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-run-systemd\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.611133 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611054 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm9gp\" (UniqueName: \"kubernetes.io/projected/e083a23b-76fd-410d-9633-ece3abc240e1-kube-api-access-dm9gp\") pod \"kas-connection-checker-685895c7c9-7jhzf\" (UID: \"e083a23b-76fd-410d-9633-ece3abc240e1\") " pod="kube-system/kas-connection-checker-685895c7c9-7jhzf" Nov 02 17:29:42.611267 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611252 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:42.611351 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611324 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-slash\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.611385 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611361 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-log-socket\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.611418 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611390 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-hostroot\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.611418 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611412 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a453427b-5025-4653-a930-ce2cf2f41cce-tmp-dir\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:42.611475 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611429 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-trusted-ca\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:42.611475 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611445 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-trusted-ca\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:42.611536 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611496 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-bound-sa-token\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:42.611568 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611527 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a453427b-5025-4653-a930-ce2cf2f41cce-tmp-dir\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:42.611568 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611531 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-etc-openvswitch\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.611624 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611576 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-sys-fs\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.611691 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611673 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jphfq\" (UniqueName: \"kubernetes.io/projected/38e5c802-99d4-4234-9e64-dee8e19b055a-kube-api-access-jphfq\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.611742 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611721 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kqjw\" (UniqueName: \"kubernetes.io/projected/a074a9ac-624c-4828-9785-c8f2aecb5827-kube-api-access-5kqjw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr\" (UID: \"a074a9ac-624c-4828-9785-c8f2aecb5827\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" Nov 02 17:29:42.611819 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611796 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc4a6db8-98e0-4387-8e64-1020978970c3-serving-cert\") pod \"service-ca-operator-544d749c8c-zbfmn\" (UID: \"bc4a6db8-98e0-4387-8e64-1020978970c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" Nov 02 17:29:42.611889 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611830 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-trusted-ca\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:42.611889 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611844 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:42.611979 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611890 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3973c00-9abb-42d0-8a2e-6e657af74f2f-ovn-node-metrics-cert\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.611979 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611925 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-trusted-ca\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:42.611979 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611925 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35669b5e-5c97-45ef-8868-4cff3fdfc07a-system-cni-dir\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:42.611979 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611965 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b4dt\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-kube-api-access-2b4dt\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:42.612181 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.611992 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b3973c00-9abb-42d0-8a2e-6e657af74f2f-ovnkube-script-lib\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.612181 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612020 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njp5v\" (UniqueName: \"kubernetes.io/projected/0eacac81-35d0-4ab1-bea4-25aaefa74c8b-kube-api-access-njp5v\") pod \"network-check-source-6f78d59f99-5fvrw\" (UID: \"0eacac81-35d0-4ab1-bea4-25aaefa74c8b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5fvrw" Nov 02 17:29:42.612181 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612049 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4de14e5-995e-4171-bb61-9f70e777688e-kubelet-config\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:42.612181 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612074 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:42.612181 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612122 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:42.612181 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612157 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/27ea270b-6e55-4652-a1bf-1431a21c3d55-ca-trust-extracted\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:42.612421 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612222 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/27ea270b-6e55-4652-a1bf-1431a21c3d55-ca-trust-extracted\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:42.612421 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612248 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bc4a6db8-98e0-4387-8e64-1020978970c3-config\") pod \"service-ca-operator-544d749c8c-zbfmn\" (UID: \"bc4a6db8-98e0-4387-8e64-1020978970c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" Nov 02 17:29:42.612421 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612269 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt4dk\" (UniqueName: \"kubernetes.io/projected/7e4dc8ce-2802-4864-aed5-affeefe41d6b-kube-api-access-pt4dk\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:42.612421 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612285 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdlkh\" (UniqueName: \"kubernetes.io/projected/4b533156-8ec1-4994-9bf6-75e4c41c3c6a-kube-api-access-zdlkh\") pod \"iptables-alerter-bmw6p\" (UID: \"4b533156-8ec1-4994-9bf6-75e4c41c3c6a\") " pod="openshift-network-operator/iptables-alerter-bmw6p" Nov 02 17:29:42.612421 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612339 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:42.612421 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612363 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f9e8821-40a8-4012-b66a-813e80d856f2-trusted-ca\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:42.612421 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612380 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-registry-certificates\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:42.612686 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612421 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-run-ovn-kubernetes\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.612686 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612469 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl4xs\" (UniqueName: \"kubernetes.io/projected/c7a1975b-4a3b-426c-977c-8f92d14810ac-kube-api-access-tl4xs\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.612686 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612502 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35669b5e-5c97-45ef-8868-4cff3fdfc07a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:42.612686 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612529 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:42.612686 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612576 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm8pr\" (UniqueName: \"kubernetes.io/projected/3d83bf78-e7b7-48bb-98be-8196e92fd320-kube-api-access-mm8pr\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:42.612686 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612607 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b3973c00-9abb-42d0-8a2e-6e657af74f2f-ovnkube-config\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.612686 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612632 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4de14e5-995e-4171-bb61-9f70e777688e-dbus\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:42.612686 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612657 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:42.612939 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612702 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw824\" (UniqueName: \"kubernetes.io/projected/bc4a6db8-98e0-4387-8e64-1020978970c3-kube-api-access-cw824\") pod \"service-ca-operator-544d749c8c-zbfmn\" (UID: \"bc4a6db8-98e0-4387-8e64-1020978970c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" Nov 02 17:29:42.612939 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612742 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-run-netns\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.612939 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612789 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-run-multus-certs\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.612939 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612819 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-var-lib-cni-bin\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.612939 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612836 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-var-lib-kubelet\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.612939 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612910 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrzfb\" (UniqueName: \"kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb\") pod \"network-check-target-qjx25\" (UID: \"97af2a01-5609-4154-a892-5de8ae3e2b9a\") " pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:42.612939 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612932 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/daf51d1a-a65f-4fad-8a82-966b8bac7ba4-konnectivity-ca\") pod \"konnectivity-agent-vs4r4\" (UID: \"daf51d1a-a65f-4fad-8a82-966b8bac7ba4\") " pod="kube-system/konnectivity-agent-vs4r4" Nov 02 17:29:42.613163 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612971 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7484d61b-5158-4729-b835-c0b372d8105b-ca-trust-extracted\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:42.613163 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.612990 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b533156-8ec1-4994-9bf6-75e4c41c3c6a-iptables-alerter-script\") pod \"iptables-alerter-bmw6p\" (UID: \"4b533156-8ec1-4994-9bf6-75e4c41c3c6a\") " pod="openshift-network-operator/iptables-alerter-bmw6p" Nov 02 17:29:42.613163 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613006 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd9gl\" (UniqueName: \"kubernetes.io/projected/704c917e-c41f-47c3-9a1e-02730361fc8a-kube-api-access-jd9gl\") pod \"ingress-canary-4qfkf\" (UID: \"704c917e-c41f-47c3-9a1e-02730361fc8a\") " pod="openshift-ingress-canary/ingress-canary-4qfkf" Nov 02 17:29:42.613163 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613021 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5f9e8821-40a8-4012-b66a-813e80d856f2-config\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:42.613163 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613038 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-default-certificate\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:42.613163 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613057 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvpxv\" (UniqueName: \"kubernetes.io/projected/b3973c00-9abb-42d0-8a2e-6e657af74f2f-kube-api-access-xvpxv\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.613163 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613087 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.613163 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613113 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-socket-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.613163 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613112 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7484d61b-5158-4729-b835-c0b372d8105b-ca-trust-extracted\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:42.613163 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613153 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wn79\" (UniqueName: \"kubernetes.io/projected/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-api-access-5wn79\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:42.613600 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613204 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-socket-dir-parent\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.613600 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613295 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-var-lib-cni-multus\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.613600 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613322 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-conf-dir\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.613600 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613342 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e4dc8ce-2802-4864-aed5-affeefe41d6b-service-ca-bundle\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:42.613600 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613359 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a453427b-5025-4653-a930-ce2cf2f41cce-config-volume\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:42.613600 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613420 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4m8s\" (UniqueName: \"kubernetes.io/projected/a453427b-5025-4653-a930-ce2cf2f41cce-kube-api-access-m4m8s\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:42.613600 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613459 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4g4\" (UniqueName: \"kubernetes.io/projected/f7a7133f-8372-49a3-8194-15ed2159aa67-kube-api-access-4g4g4\") pod \"volume-data-source-validator-58c9f5f46d-dmr89\" (UID: \"f7a7133f-8372-49a3-8194-15ed2159aa67\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dmr89" Nov 02 17:29:42.613600 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613486 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8psf\" (UniqueName: \"kubernetes.io/projected/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-kube-api-access-n8psf\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:42.613600 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613504 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-image-registry-private-configuration\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:42.613600 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613520 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-systemd-units\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.613600 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613536 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-cni-dir\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.613600 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613557 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-installation-pull-secrets\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:42.613600 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613564 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e4dc8ce-2802-4864-aed5-affeefe41d6b-service-ca-bundle\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:42.613600 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613582 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b44f96b4-5c43-40cc-9c89-63d4ec368e8b-serviceca\") pod \"node-ca-jxsgc\" (UID: \"b44f96b4-5c43-40cc-9c89-63d4ec368e8b\") " pod="openshift-image-registry/node-ca-jxsgc" Nov 02 17:29:42.614140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613621 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-installation-pull-secrets\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:42.614140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613655 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.614140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613798 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-run\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.614140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613822 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-image-registry-private-configuration\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:42.614140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613872 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-installation-pull-secrets\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:42.614140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613878 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-var-lib-kubelet\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.614140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613873 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-installation-pull-secrets\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:42.614140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613912 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z95xd\" (UniqueName: \"kubernetes.io/projected/9368ef45-b865-4cdf-95a6-5088fc54277e-kube-api-access-z95xd\") pod \"node-resolver-s6s58\" (UID: \"9368ef45-b865-4cdf-95a6-5088fc54277e\") " pod="openshift-dns/node-resolver-s6s58" Nov 02 17:29:42.614140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613952 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:42.614140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.613983 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-certificates\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:42.614140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614018 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b44f96b4-5c43-40cc-9c89-63d4ec368e8b-host\") pod \"node-ca-jxsgc\" (UID: \"b44f96b4-5c43-40cc-9c89-63d4ec368e8b\") " pod="openshift-image-registry/node-ca-jxsgc" Nov 02 17:29:42.614140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614041 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7e4dc8ce-2802-4864-aed5-affeefe41d6b-snapshots\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:42.614140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614059 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35669b5e-5c97-45ef-8868-4cff3fdfc07a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:42.614140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614113 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38e5c802-99d4-4234-9e64-dee8e19b055a-tmp\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.614140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614134 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7e4dc8ce-2802-4864-aed5-affeefe41d6b-snapshots\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:42.614140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614143 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-cnibin\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.614602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614163 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jf4x\" (UniqueName: \"kubernetes.io/projected/42f9e736-411e-46a2-89d0-2a3870e2f95a-kube-api-access-8jf4x\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:42.614602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614183 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35669b5e-5c97-45ef-8868-4cff3fdfc07a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:42.614602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614224 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-sys\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.614602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614229 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38e5c802-99d4-4234-9e64-dee8e19b055a-tmp\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.614602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614268 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:42.614602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614308 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm4fn\" (UniqueName: \"kubernetes.io/projected/b44f96b4-5c43-40cc-9c89-63d4ec368e8b-kube-api-access-xm4fn\") pod \"node-ca-jxsgc\" (UID: \"b44f96b4-5c43-40cc-9c89-63d4ec368e8b\") " pod="openshift-image-registry/node-ca-jxsgc" Nov 02 17:29:42.614602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614343 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-cni-bin\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.614602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614364 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqrgs\" (UniqueName: \"kubernetes.io/projected/46d2456f-62b2-4aa5-b172-1c28ab7615ac-kube-api-access-wqrgs\") pod \"kas-connection-checker-685895c7c9-nxnkz\" (UID: \"46d2456f-62b2-4aa5-b172-1c28ab7615ac\") " pod="kube-system/kas-connection-checker-685895c7c9-nxnkz" Nov 02 17:29:42.614602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614380 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b533156-8ec1-4994-9bf6-75e4c41c3c6a-host-slash\") pod \"iptables-alerter-bmw6p\" (UID: \"4b533156-8ec1-4994-9bf6-75e4c41c3c6a\") " pod="openshift-network-operator/iptables-alerter-bmw6p" Nov 02 17:29:42.614602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614411 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:42.614602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614487 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-run-k8s-cni-cncf-io\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.614602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614512 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-kubernetes\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.614602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614537 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-host\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.614602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614571 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/42f9e736-411e-46a2-89d0-2a3870e2f95a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:42.614602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614596 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tdtz\" (UniqueName: \"kubernetes.io/projected/f20b07b4-cd27-4410-abb2-c0bd50c01269-kube-api-access-9tdtz\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:42.615035 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614762 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.615035 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614783 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9368ef45-b865-4cdf-95a6-5088fc54277e-hosts-file\") pod \"node-resolver-s6s58\" (UID: \"9368ef45-b865-4cdf-95a6-5088fc54277e\") " pod="openshift-dns/node-resolver-s6s58" Nov 02 17:29:42.615035 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614803 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spg94\" (UniqueName: \"kubernetes.io/projected/5f9e8821-40a8-4012-b66a-813e80d856f2-kube-api-access-spg94\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:42.615035 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614829 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:42.615035 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614886 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e4dc8ce-2802-4864-aed5-affeefe41d6b-tmp\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:42.615035 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614914 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e4dc8ce-2802-4864-aed5-affeefe41d6b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:42.615035 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614932 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-sysconfig\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.615035 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.614991 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-etc-kubernetes\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.615035 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615020 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/72445bf5-a7a3-49dc-9b97-5589ae901032-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7scpf\" (UID: \"72445bf5-a7a3-49dc-9b97-5589ae901032\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:42.615270 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615040 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e4dc8ce-2802-4864-aed5-affeefe41d6b-tmp\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:42.615270 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615038 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-kubelet\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.615270 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615084 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-run-openvswitch\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.615270 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615105 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-tuned\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.615270 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615154 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-run-netns\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.615270 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615169 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-tuned\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.615270 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615172 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-node-log\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.615270 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615210 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b3973c00-9abb-42d0-8a2e-6e657af74f2f-env-overrides\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.615270 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615248 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-systemd\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.615562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615277 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9368ef45-b865-4cdf-95a6-5088fc54277e-tmp-dir\") pod \"node-resolver-s6s58\" (UID: \"9368ef45-b865-4cdf-95a6-5088fc54277e\") " pod="openshift-dns/node-resolver-s6s58" Nov 02 17:29:42.615562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615297 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f9e8821-40a8-4012-b66a-813e80d856f2-serving-cert\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:42.615562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615331 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-cni-netd\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.615562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615351 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfp9\" (UniqueName: \"kubernetes.io/projected/fbbf8db7-5d42-47b6-b03f-dcd4b68f3211-kube-api-access-slfp9\") pod \"kas-connection-checker-685895c7c9-hxpgv\" (UID: \"fbbf8db7-5d42-47b6-b03f-dcd4b68f3211\") " pod="kube-system/kas-connection-checker-685895c7c9-hxpgv" Nov 02 17:29:42.615562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615366 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9368ef45-b865-4cdf-95a6-5088fc54277e-tmp-dir\") pod \"node-resolver-s6s58\" (UID: \"9368ef45-b865-4cdf-95a6-5088fc54277e\") " pod="openshift-dns/node-resolver-s6s58" Nov 02 17:29:42.615562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615370 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a074a9ac-624c-4828-9785-c8f2aecb5827-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr\" (UID: \"a074a9ac-624c-4828-9785-c8f2aecb5827\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" Nov 02 17:29:42.615562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615387 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-var-lib-openvswitch\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.615562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615402 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-device-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.615562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615421 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrvfr\" (UniqueName: \"kubernetes.io/projected/35669b5e-5c97-45ef-8868-4cff3fdfc07a-kube-api-access-lrvfr\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:42.615562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615437 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-sysctl-conf\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.615562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615453 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-lib-modules\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.615562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615480 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4b9af78-fe26-40bc-a4df-126ffd0885d2-cni-binary-copy\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.615562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615499 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7scpf\" (UID: \"72445bf5-a7a3-49dc-9b97-5589ae901032\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:42.615562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615533 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-crio-socket\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:42.615562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615565 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-data-volume\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:42.616013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615603 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a074a9ac-624c-4828-9785-c8f2aecb5827-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr\" (UID: \"a074a9ac-624c-4828-9785-c8f2aecb5827\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" Nov 02 17:29:42.616013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615634 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-data-volume\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:42.616013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615638 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-image-registry-private-configuration\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:42.616013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615717 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-daemon-config\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.616013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615758 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/704c917e-c41f-47c3-9a1e-02730361fc8a-cert\") pod \"ingress-canary-4qfkf\" (UID: \"704c917e-c41f-47c3-9a1e-02730361fc8a\") " pod="openshift-ingress-canary/ingress-canary-4qfkf" Nov 02 17:29:42.616013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615772 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-image-registry-private-configuration\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:42.616013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615786 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:42.616013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615816 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpdh7\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-kube-api-access-rpdh7\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:42.616013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615844 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-stats-auth\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:42.616013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615879 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-run-ovn\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.616013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615898 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7e4dc8ce-2802-4864-aed5-affeefe41d6b-serving-cert\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:42.616013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615913 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-registration-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.616013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615939 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35669b5e-5c97-45ef-8868-4cff3fdfc07a-cni-binary-copy\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:42.616013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.615965 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-modprobe-d\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.616013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.616015 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-sysctl-d\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.616421 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.616035 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-system-cni-dir\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.627567 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.627544 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 17:29:42.631253 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.631228 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 17:29:42.631375 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.631264 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 17:29:42.631716 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.631700 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 17:29:42.631754 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.631722 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 17:29:42.631982 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.631967 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 17:29:42.632034 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.631995 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 17:29:42.632529 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.632515 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 17:29:42.632590 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.632543 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 17:29:42.632590 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.632545 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 17:29:42.633686 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.633669 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a453427b-5025-4653-a930-ce2cf2f41cce-config-volume\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:42.633762 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.633692 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:42.633936 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.633920 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b44f96b4-5c43-40cc-9c89-63d4ec368e8b-serviceca\") pod \"node-ca-jxsgc\" (UID: \"b44f96b4-5c43-40cc-9c89-63d4ec368e8b\") " pod="openshift-image-registry/node-ca-jxsgc" Nov 02 17:29:42.633996 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.633959 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 17:29:42.633996 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.633968 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 17:29:42.633996 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.633973 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 17:29:42.634203 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.634180 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 17:29:42.634264 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.634241 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 17:29:42.634310 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.634268 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-certificates\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:42.634496 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.634482 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 17:29:42.636428 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.636412 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 17:29:42.636504 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.636452 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4b9af78-fe26-40bc-a4df-126ffd0885d2-cni-binary-copy\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.636504 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.636501 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 17:29:42.636599 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.636501 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35669b5e-5c97-45ef-8868-4cff3fdfc07a-cni-binary-copy\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:42.636644 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.636607 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 17:29:42.636644 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.636628 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 17:29:42.636732 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.636659 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 17:29:42.636732 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.636686 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5nstj\"" Nov 02 17:29:42.636836 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.636811 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 17:29:42.636953 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.636835 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 17:29:42.636953 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.636886 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 17:29:42.636953 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.636893 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 17:29:42.637254 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.637229 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 17:29:42.637336 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.637318 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 17:29:42.638035 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.637845 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:42.638035 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.637916 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 17:29:42.638035 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.638027 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 17:29:42.638216 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.638187 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ncmj6\"" Nov 02 17:29:42.639211 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.638750 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 17:29:42.639211 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.638912 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 17:29:42.639211 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.639182 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 17:29:42.639211 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.639198 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 17:29:42.639541 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.639300 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 17:29:42.639612 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.639538 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 17:29:42.639612 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.639550 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 17:29:42.639795 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.639777 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 17:29:42.639872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.639841 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 17:29:42.639872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.639786 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:42.639968 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.639952 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 17:29:42.640015 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.639987 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:42.640070 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.640015 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 17:29:42.640221 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.640203 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 17:29:42.640283 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.640240 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 17:29:42.640335 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.640304 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9bnd5\"" Nov 02 17:29:42.640455 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.640442 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 17:29:42.640455 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.640450 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:42.640758 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.640730 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xdl26\"" Nov 02 17:29:42.641986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.641157 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wbstr\"" Nov 02 17:29:42.641986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.641257 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9d6qd\"" Nov 02 17:29:42.641986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.641427 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 17:29:42.641986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.641573 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 17:29:42.641986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.641649 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 17:29:42.641986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.641748 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 17:29:42.642341 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.642132 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc4a6db8-98e0-4387-8e64-1020978970c3-serving-cert\") pod \"service-ca-operator-544d749c8c-zbfmn\" (UID: \"bc4a6db8-98e0-4387-8e64-1020978970c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" Nov 02 17:29:42.642341 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.642169 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:42.642428 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.642385 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3973c00-9abb-42d0-8a2e-6e657af74f2f-ovn-node-metrics-cert\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.642682 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.642625 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bc4a6db8-98e0-4387-8e64-1020978970c3-config\") pod \"service-ca-operator-544d749c8c-zbfmn\" (UID: \"bc4a6db8-98e0-4387-8e64-1020978970c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" Nov 02 17:29:42.642682 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.642650 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-registry-certificates\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:42.642836 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.642689 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 17:29:42.643313 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.643296 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/daf51d1a-a65f-4fad-8a82-966b8bac7ba4-konnectivity-ca\") pod \"konnectivity-agent-vs4r4\" (UID: \"daf51d1a-a65f-4fad-8a82-966b8bac7ba4\") " pod="kube-system/konnectivity-agent-vs4r4" Nov 02 17:29:42.643375 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.643334 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5f9e8821-40a8-4012-b66a-813e80d856f2-config\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:42.643375 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.643334 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35669b5e-5c97-45ef-8868-4cff3fdfc07a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:42.643442 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.643364 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-default-certificate\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:42.645284 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.645264 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/72445bf5-a7a3-49dc-9b97-5589ae901032-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7scpf\" (UID: \"72445bf5-a7a3-49dc-9b97-5589ae901032\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:42.645347 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.645264 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35669b5e-5c97-45ef-8868-4cff3fdfc07a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:42.645514 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.645496 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f9e8821-40a8-4012-b66a-813e80d856f2-serving-cert\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:42.645657 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.645644 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a074a9ac-624c-4828-9785-c8f2aecb5827-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr\" (UID: \"a074a9ac-624c-4828-9785-c8f2aecb5827\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" Nov 02 17:29:42.646652 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.646635 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7e4dc8ce-2802-4864-aed5-affeefe41d6b-serving-cert\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:42.646728 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.646667 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a074a9ac-624c-4828-9785-c8f2aecb5827-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr\" (UID: \"a074a9ac-624c-4828-9785-c8f2aecb5827\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" Nov 02 17:29:42.646728 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.646685 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-stats-auth\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:42.646728 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.646710 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/704c917e-c41f-47c3-9a1e-02730361fc8a-cert\") pod \"ingress-canary-4qfkf\" (UID: \"704c917e-c41f-47c3-9a1e-02730361fc8a\") " pod="openshift-ingress-canary/ingress-canary-4qfkf" Nov 02 17:29:42.661121 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661102 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 17:29:42.661317 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661300 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 17:29:42.661362 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661313 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jl8vw\"" Nov 02 17:29:42.661362 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661315 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:29:42.661548 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661532 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t2hrp\"" Nov 02 17:29:42.661601 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661569 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ppwdb\"" Nov 02 17:29:42.661726 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661711 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 17:29:42.661790 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661752 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 17:29:42.661790 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661776 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-qz666\"" Nov 02 17:29:42.661910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661790 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bl7zd\"" Nov 02 17:29:42.661910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661781 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 17:29:42.661910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661842 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 17:29:42.661910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661845 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 17:29:42.661910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661805 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zt2z5\"" Nov 02 17:29:42.661910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661875 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mwsx7\"" Nov 02 17:29:42.661910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661874 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 17:29:42.661910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661779 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 17:29:42.661910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661815 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-npw6n\"" Nov 02 17:29:42.661910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661891 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 17:29:42.662358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661928 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-chwwq\"" Nov 02 17:29:42.662358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.661966 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4b9fl\"" Nov 02 17:29:42.662358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.662106 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 17:29:42.662358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.662203 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fw7vt\"" Nov 02 17:29:42.662358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.662229 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rj9zp\"" Nov 02 17:29:42.662358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.662272 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 17:29:42.662358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.662325 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nn2p2\"" Nov 02 17:29:42.662655 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.662377 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-q8x9s\"" Nov 02 17:29:42.662655 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.662409 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 17:29:42.662655 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.662436 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9lngf\"" Nov 02 17:29:42.662655 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.662527 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qbqb7\"" Nov 02 17:29:42.662655 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.662543 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw824\" (UniqueName: \"kubernetes.io/projected/bc4a6db8-98e0-4387-8e64-1020978970c3-kube-api-access-cw824\") pod \"service-ca-operator-544d749c8c-zbfmn\" (UID: \"bc4a6db8-98e0-4387-8e64-1020978970c3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" Nov 02 17:29:42.662806 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.662689 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f9e8821-40a8-4012-b66a-813e80d856f2-trusted-ca\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:42.662844 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.662830 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b3973c00-9abb-42d0-8a2e-6e657af74f2f-ovnkube-config\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.662910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.662890 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt4dk\" (UniqueName: \"kubernetes.io/projected/7e4dc8ce-2802-4864-aed5-affeefe41d6b-kube-api-access-pt4dk\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:42.662943 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.662916 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl4xs\" (UniqueName: \"kubernetes.io/projected/c7a1975b-4a3b-426c-977c-8f92d14810ac-kube-api-access-tl4xs\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.663530 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.663517 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b533156-8ec1-4994-9bf6-75e4c41c3c6a-iptables-alerter-script\") pod \"iptables-alerter-bmw6p\" (UID: \"4b533156-8ec1-4994-9bf6-75e4c41c3c6a\") " pod="openshift-network-operator/iptables-alerter-bmw6p" Nov 02 17:29:42.665595 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.665579 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b3973c00-9abb-42d0-8a2e-6e657af74f2f-env-overrides\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.665816 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.665799 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e4dc8ce-2802-4864-aed5-affeefe41d6b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tszgp\" (UID: \"7e4dc8ce-2802-4864-aed5-affeefe41d6b\") " pod="openshift-insights/insights-operator-7c4986d449-tszgp" Nov 02 17:29:42.665966 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.665951 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-daemon-config\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.667999 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.667984 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4m8s\" (UniqueName: \"kubernetes.io/projected/a453427b-5025-4653-a930-ce2cf2f41cce-kube-api-access-m4m8s\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:42.668071 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.668055 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd9gl\" (UniqueName: \"kubernetes.io/projected/704c917e-c41f-47c3-9a1e-02730361fc8a-kube-api-access-jd9gl\") pod \"ingress-canary-4qfkf\" (UID: \"704c917e-c41f-47c3-9a1e-02730361fc8a\") " pod="openshift-ingress-canary/ingress-canary-4qfkf" Nov 02 17:29:42.668108 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.668083 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrzfb\" (UniqueName: \"kubernetes.io/projected/97af2a01-5609-4154-a892-5de8ae3e2b9a-kube-api-access-vrzfb\") pod \"network-check-target-qjx25\" (UID: \"97af2a01-5609-4154-a892-5de8ae3e2b9a\") " pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:42.670638 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.670618 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvpxv\" (UniqueName: \"kubernetes.io/projected/b3973c00-9abb-42d0-8a2e-6e657af74f2f-kube-api-access-xvpxv\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.671009 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.670996 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 17:29:42.671134 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.671118 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wn79\" (UniqueName: \"kubernetes.io/projected/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-api-access-5wn79\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:42.671305 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.671293 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/daf51d1a-a65f-4fad-8a82-966b8bac7ba4-agent-certs\") pod \"konnectivity-agent-vs4r4\" (UID: \"daf51d1a-a65f-4fad-8a82-966b8bac7ba4\") " pod="kube-system/konnectivity-agent-vs4r4" Nov 02 17:29:42.671379 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.671364 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tdtz\" (UniqueName: \"kubernetes.io/projected/f20b07b4-cd27-4410-abb2-c0bd50c01269-kube-api-access-9tdtz\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:42.671900 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.671886 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 17:29:42.672233 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.672218 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b3973c00-9abb-42d0-8a2e-6e657af74f2f-ovnkube-script-lib\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.672326 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.672312 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vjnnw\"" Nov 02 17:29:42.672358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.672342 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 17:29:42.675011 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.674994 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm8pr\" (UniqueName: \"kubernetes.io/projected/3d83bf78-e7b7-48bb-98be-8196e92fd320-kube-api-access-mm8pr\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:42.675214 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.675199 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4de14e5-995e-4171-bb61-9f70e777688e-original-pull-secret\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:42.675246 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.675220 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/42f9e736-411e-46a2-89d0-2a3870e2f95a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:42.676390 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.676374 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4g4\" (UniqueName: \"kubernetes.io/projected/f7a7133f-8372-49a3-8194-15ed2159aa67-kube-api-access-4g4g4\") pod \"volume-data-source-validator-58c9f5f46d-dmr89\" (UID: \"f7a7133f-8372-49a3-8194-15ed2159aa67\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dmr89" Nov 02 17:29:42.682349 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.682333 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdlkh\" (UniqueName: \"kubernetes.io/projected/4b533156-8ec1-4994-9bf6-75e4c41c3c6a-kube-api-access-zdlkh\") pod \"iptables-alerter-bmw6p\" (UID: \"4b533156-8ec1-4994-9bf6-75e4c41c3c6a\") " pod="openshift-network-operator/iptables-alerter-bmw6p" Nov 02 17:29:42.686970 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.686946 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrvfr\" (UniqueName: \"kubernetes.io/projected/35669b5e-5c97-45ef-8868-4cff3fdfc07a-kube-api-access-lrvfr\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:42.688315 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.688278 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8psf\" (UniqueName: \"kubernetes.io/projected/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-kube-api-access-n8psf\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:42.688605 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.688589 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z95xd\" (UniqueName: \"kubernetes.io/projected/9368ef45-b865-4cdf-95a6-5088fc54277e-kube-api-access-z95xd\") pod \"node-resolver-s6s58\" (UID: \"9368ef45-b865-4cdf-95a6-5088fc54277e\") " pod="openshift-dns/node-resolver-s6s58" Nov 02 17:29:42.692491 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.692475 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfjxf\" (UniqueName: \"kubernetes.io/projected/a4b9af78-fe26-40bc-a4df-126ffd0885d2-kube-api-access-mfjxf\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.693601 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.693583 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spg94\" (UniqueName: \"kubernetes.io/projected/5f9e8821-40a8-4012-b66a-813e80d856f2-kube-api-access-spg94\") pod \"console-operator-7cd56b8dbc-xqzqj\" (UID: \"5f9e8821-40a8-4012-b66a-813e80d856f2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:42.694116 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.694068 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-bound-sa-token\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:42.694329 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.694315 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b4dt\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-kube-api-access-2b4dt\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:42.695572 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.695556 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-bound-sa-token\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:42.695613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.695557 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm9gp\" (UniqueName: \"kubernetes.io/projected/e083a23b-76fd-410d-9633-ece3abc240e1-kube-api-access-dm9gp\") pod \"kas-connection-checker-685895c7c9-7jhzf\" (UID: \"e083a23b-76fd-410d-9633-ece3abc240e1\") " pod="kube-system/kas-connection-checker-685895c7c9-7jhzf" Nov 02 17:29:42.696595 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.696578 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njp5v\" (UniqueName: \"kubernetes.io/projected/0eacac81-35d0-4ab1-bea4-25aaefa74c8b-kube-api-access-njp5v\") pod \"network-check-source-6f78d59f99-5fvrw\" (UID: \"0eacac81-35d0-4ab1-bea4-25aaefa74c8b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5fvrw" Nov 02 17:29:42.697755 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.697731 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jphfq\" (UniqueName: \"kubernetes.io/projected/38e5c802-99d4-4234-9e64-dee8e19b055a-kube-api-access-jphfq\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.698017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.698002 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slfp9\" (UniqueName: \"kubernetes.io/projected/fbbf8db7-5d42-47b6-b03f-dcd4b68f3211-kube-api-access-slfp9\") pod \"kas-connection-checker-685895c7c9-hxpgv\" (UID: \"fbbf8db7-5d42-47b6-b03f-dcd4b68f3211\") " pod="kube-system/kas-connection-checker-685895c7c9-hxpgv" Nov 02 17:29:42.698094 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.698078 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kqjw\" (UniqueName: \"kubernetes.io/projected/a074a9ac-624c-4828-9785-c8f2aecb5827-kube-api-access-5kqjw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr\" (UID: \"a074a9ac-624c-4828-9785-c8f2aecb5827\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" Nov 02 17:29:42.698301 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.698286 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jf4x\" (UniqueName: \"kubernetes.io/projected/42f9e736-411e-46a2-89d0-2a3870e2f95a-kube-api-access-8jf4x\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:42.698782 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.698767 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpdh7\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-kube-api-access-rpdh7\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:42.698894 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.698879 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm4fn\" (UniqueName: \"kubernetes.io/projected/b44f96b4-5c43-40cc-9c89-63d4ec368e8b-kube-api-access-xm4fn\") pod \"node-ca-jxsgc\" (UID: \"b44f96b4-5c43-40cc-9c89-63d4ec368e8b\") " pod="openshift-image-registry/node-ca-jxsgc" Nov 02 17:29:42.701024 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.701008 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqrgs\" (UniqueName: \"kubernetes.io/projected/46d2456f-62b2-4aa5-b172-1c28ab7615ac-kube-api-access-wqrgs\") pod \"kas-connection-checker-685895c7c9-nxnkz\" (UID: \"46d2456f-62b2-4aa5-b172-1c28ab7615ac\") " pod="kube-system/kas-connection-checker-685895c7c9-nxnkz" Nov 02 17:29:42.716444 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716425 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b533156-8ec1-4994-9bf6-75e4c41c3c6a-host-slash\") pod \"iptables-alerter-bmw6p\" (UID: \"4b533156-8ec1-4994-9bf6-75e4c41c3c6a\") " pod="openshift-network-operator/iptables-alerter-bmw6p" Nov 02 17:29:42.716528 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716451 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-run-k8s-cni-cncf-io\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.716528 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716473 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-kubernetes\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.716528 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716487 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-host\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.716528 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716504 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.716671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716526 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9368ef45-b865-4cdf-95a6-5088fc54277e-hosts-file\") pod \"node-resolver-s6s58\" (UID: \"9368ef45-b865-4cdf-95a6-5088fc54277e\") " pod="openshift-dns/node-resolver-s6s58" Nov 02 17:29:42.716671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716540 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b533156-8ec1-4994-9bf6-75e4c41c3c6a-host-slash\") pod \"iptables-alerter-bmw6p\" (UID: \"4b533156-8ec1-4994-9bf6-75e4c41c3c6a\") " pod="openshift-network-operator/iptables-alerter-bmw6p" Nov 02 17:29:42.716671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716554 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:42.716671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716568 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-host\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.716671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716581 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.716671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716571 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-run-k8s-cni-cncf-io\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.716671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716587 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9368ef45-b865-4cdf-95a6-5088fc54277e-hosts-file\") pod \"node-resolver-s6s58\" (UID: \"9368ef45-b865-4cdf-95a6-5088fc54277e\") " pod="openshift-dns/node-resolver-s6s58" Nov 02 17:29:42.716671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716594 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-sysconfig\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.716671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716636 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-sysconfig\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.716671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716643 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-etc-kubernetes\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.716671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716547 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-kubernetes\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.716671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716669 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-kubelet\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716693 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-kubelet\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716699 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-etc-kubernetes\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.716699 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle podName:3d83bf78-e7b7-48bb-98be-8196e92fd320 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:43.216678814 +0000 UTC m=+1.838038589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle") pod "router-default-79c77b5cd5-c6svb" (UID: "3d83bf78-e7b7-48bb-98be-8196e92fd320") : configmap references non-existent config key: service-ca.crt Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716729 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-run-openvswitch\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716747 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-run-netns\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716769 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-node-log\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716773 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-run-openvswitch\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716787 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-systemd\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716803 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-run-netns\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716804 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-cni-netd\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716821 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-cni-netd\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716840 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-var-lib-openvswitch\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716865 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-node-log\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716881 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-device-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716884 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-systemd\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716910 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-var-lib-openvswitch\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716932 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-device-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716953 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-sysctl-conf\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716968 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-lib-modules\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.716985 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7scpf\" (UID: \"72445bf5-a7a3-49dc-9b97-5589ae901032\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717002 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-crio-socket\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717020 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717037 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-run-ovn\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717060 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-crio-socket\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717060 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-registration-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717100 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-modprobe-d\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717105 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-registration-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717117 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-lib-modules\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.717111 4860 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717131 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-run-ovn\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717147 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-sysctl-d\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717150 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-sysctl-conf\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.717128 4860 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.717190 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert podName:72445bf5-a7a3-49dc-9b97-5589ae901032 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:43.217174471 +0000 UTC m=+1.838534236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7scpf" (UID: "72445bf5-a7a3-49dc-9b97-5589ae901032") : secret "networking-console-plugin-cert" not found Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717192 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-modprobe-d\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.717200 4860 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8ffbdd9-xqf4t: secret "image-registry-tls" not found Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717214 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-system-cni-dir\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717240 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-etc-sysctl-d\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717181 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-system-cni-dir\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.717261 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.717267 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls podName:27ea270b-6e55-4652-a1bf-1431a21c3d55 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:43.217250833 +0000 UTC m=+1.838610603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls") pod "image-registry-6f8ffbdd9-xqf4t" (UID: "27ea270b-6e55-4652-a1bf-1431a21c3d55") : secret "image-registry-tls" not found Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717305 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35669b5e-5c97-45ef-8868-4cff3fdfc07a-cnibin\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717336 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-run-systemd\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717351 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35669b5e-5c97-45ef-8868-4cff3fdfc07a-cnibin\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717362 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-slash\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717395 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-log-socket\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717413 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-run-systemd\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717418 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-hostroot\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717445 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-etc-openvswitch\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717452 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-slash\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717471 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-sys-fs\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717481 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-hostroot\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717504 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-log-socket\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717503 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717535 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35669b5e-5c97-45ef-8868-4cff3fdfc07a-system-cni-dir\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717555 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4de14e5-995e-4171-bb61-9f70e777688e-kubelet-config\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717562 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-etc-openvswitch\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717571 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717588 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717612 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717623 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-sys-fs\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717629 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-run-ovn-kubernetes\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717649 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-run-ovn-kubernetes\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717659 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717687 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4de14e5-995e-4171-bb61-9f70e777688e-dbus\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.717706 4860 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717719 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717742 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4de14e5-995e-4171-bb61-9f70e777688e-kubelet-config\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.717756 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls podName:f20b07b4-cd27-4410-abb2-c0bd50c01269 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:43.217743026 +0000 UTC m=+1.839102794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dxkqn" (UID: "f20b07b4-cd27-4410-abb2-c0bd50c01269") : secret "samples-operator-tls" not found Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.717759 4860 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.717770 4860 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717773 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-run-netns\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.717798 4860 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717800 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-run-multus-certs\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.717817 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls podName:42f9e736-411e-46a2-89d0-2a3870e2f95a nodeName:}" failed. No retries permitted until 2026-11-02 17:29:43.217802337 +0000 UTC m=+1.839162094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pthgk" (UID: "42f9e736-411e-46a2-89d0-2a3870e2f95a") : secret "cluster-monitoring-operator-tls" not found Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.717771 4860 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78749f569b-zzhxm: secret "image-registry-tls" not found Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717833 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-run-multus-certs\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717859 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-run-netns\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717843 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-var-lib-cni-bin\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.717878 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-rbac-proxy-cm podName:85cbfa4a-71cd-4b2c-8596-a484deeb15a5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:43.217843181 +0000 UTC m=+1.839202945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fbrw7" (UID: "85cbfa4a-71cd-4b2c-8596-a484deeb15a5") : configmap "kube-rbac-proxy" not found Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717894 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-var-lib-cni-bin\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.717912 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls podName:7484d61b-5158-4729-b835-c0b372d8105b nodeName:}" failed. No retries permitted until 2026-11-02 17:29:43.217903024 +0000 UTC m=+1.839262787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls") pod "image-registry-78749f569b-zzhxm" (UID: "7484d61b-5158-4729-b835-c0b372d8105b") : secret "image-registry-tls" not found Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.717919 4860 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717932 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-var-lib-kubelet\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.717951 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs podName:3d83bf78-e7b7-48bb-98be-8196e92fd320 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:43.217941377 +0000 UTC m=+1.839301133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs") pod "router-default-79c77b5cd5-c6svb" (UID: "3d83bf78-e7b7-48bb-98be-8196e92fd320") : secret "router-metrics-certs-default" not found Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717959 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-var-lib-kubelet\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.717988 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718005 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-socket-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.718008 4860 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.718055 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls podName:85cbfa4a-71cd-4b2c-8596-a484deeb15a5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:43.218040526 +0000 UTC m=+1.839400316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fbrw7" (UID: "85cbfa4a-71cd-4b2c-8596-a484deeb15a5") : secret "insights-runtime-extractor-tls" not found Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718056 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-socket-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718069 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4de14e5-995e-4171-bb61-9f70e777688e-dbus\") pod \"global-pull-secret-syncer-w8vvj\" (UID: \"b4de14e5-995e-4171-bb61-9f70e777688e\") " pod="kube-system/global-pull-secret-syncer-w8vvj" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718082 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-socket-dir-parent\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718106 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-var-lib-cni-multus\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718110 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35669b5e-5c97-45ef-8868-4cff3fdfc07a-system-cni-dir\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718109 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718133 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-socket-dir-parent\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718130 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-conf-dir\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718149 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-host-var-lib-cni-multus\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718161 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-conf-dir\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718165 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-systemd-units\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718187 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-cni-dir\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718205 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718219 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-run\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718232 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-var-lib-kubelet\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718234 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-systemd-units\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718257 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718263 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-multus-cni-dir\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718278 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b44f96b4-5c43-40cc-9c89-63d4ec368e8b-host\") pod \"node-ca-jxsgc\" (UID: \"b44f96b4-5c43-40cc-9c89-63d4ec368e8b\") " pod="openshift-image-registry/node-ca-jxsgc" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718281 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7a1975b-4a3b-426c-977c-8f92d14810ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-z8jfr\" (UID: \"c7a1975b-4a3b-426c-977c-8f92d14810ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8jfr" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718287 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-run\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718285 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-var-lib-kubelet\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718305 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b44f96b4-5c43-40cc-9c89-63d4ec368e8b-host\") pod \"node-ca-jxsgc\" (UID: \"b44f96b4-5c43-40cc-9c89-63d4ec368e8b\") " pod="openshift-image-registry/node-ca-jxsgc" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.718356 4860 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.718390 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs podName:4b5c29e0-6bfd-4f43-867a-c460dc227fbc nodeName:}" failed. No retries permitted until 2026-11-02 17:29:43.218374991 +0000 UTC m=+1.839734745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs") pod "network-metrics-daemon-g6b2c" (UID: "4b5c29e0-6bfd-4f43-867a-c460dc227fbc") : secret "metrics-daemon-secret" not found Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718401 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35669b5e-5c97-45ef-8868-4cff3fdfc07a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718420 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-cnibin\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718436 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-sys\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718450 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718458 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35669b5e-5c97-45ef-8868-4cff3fdfc07a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p6d66\" (UID: \"35669b5e-5c97-45ef-8868-4cff3fdfc07a\") " pod="openshift-multus/multus-additional-cni-plugins-p6d66" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718466 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-cni-bin\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718497 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3973c00-9abb-42d0-8a2e-6e657af74f2f-host-cni-bin\") pod \"ovnkube-node-k58q5\" (UID: \"b3973c00-9abb-42d0-8a2e-6e657af74f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718505 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4b9af78-fe26-40bc-a4df-126ffd0885d2-cnibin\") pod \"multus-h6dft\" (UID: \"a4b9af78-fe26-40bc-a4df-126ffd0885d2\") " pod="openshift-multus/multus-h6dft" Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.718533 4860 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:42.718571 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls podName:a453427b-5025-4653-a930-ce2cf2f41cce nodeName:}" failed. No retries permitted until 2026-11-02 17:29:43.218558585 +0000 UTC m=+1.839918359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls") pod "dns-default-w7zsl" (UID: "a453427b-5025-4653-a930-ce2cf2f41cce") : secret "dns-default-metrics-tls" not found Nov 02 17:29:42.718826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.718588 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38e5c802-99d4-4234-9e64-dee8e19b055a-sys\") pod \"tuned-ptdb2\" (UID: \"38e5c802-99d4-4234-9e64-dee8e19b055a\") " pod="openshift-cluster-node-tuning-operator/tuned-ptdb2" Nov 02 17:29:42.802187 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:42.802153 4860 scope.go:122] "RemoveContainer" containerID="3d9ba56bcbd5f834961a29159d8acb53f61960759a915226ffb42f05b8a1ea33" Nov 02 17:29:43.223703 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:43.223614 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:43.223703 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:43.223652 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:43.223703 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:43.223669 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:43.223703 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:43.223692 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:43.224005 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:43.223744 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:43.224005 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:43.223779 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:43.224005 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:43.223827 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:43.224005 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.223778 4860 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 17:29:43.224005 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:43.223875 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:43.224005 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.223893 4860 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 17:29:43.224005 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.223902 4860 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 17:29:43.224005 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.223918 4860 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78749f569b-zzhxm: secret "image-registry-tls" not found Nov 02 17:29:43.224005 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.223936 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls podName:f20b07b4-cd27-4410-abb2-c0bd50c01269 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:44.223918277 +0000 UTC m=+2.845278049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dxkqn" (UID: "f20b07b4-cd27-4410-abb2-c0bd50c01269") : secret "samples-operator-tls" not found Nov 02 17:29:43.224005 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:43.223916 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:43.224005 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.223964 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs podName:4b5c29e0-6bfd-4f43-867a-c460dc227fbc nodeName:}" failed. No retries permitted until 2026-11-02 17:29:44.223946056 +0000 UTC m=+2.845305838 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs") pod "network-metrics-daemon-g6b2c" (UID: "4b5c29e0-6bfd-4f43-867a-c460dc227fbc") : secret "metrics-daemon-secret" not found Nov 02 17:29:43.224005 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.223971 4860 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 17:29:43.224005 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.223789 4860 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 17:29:43.224631 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.223789 4860 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 17:29:43.224631 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.223825 4860 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 17:29:43.224631 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.223988 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls podName:7484d61b-5158-4729-b835-c0b372d8105b nodeName:}" failed. No retries permitted until 2026-11-02 17:29:44.223978163 +0000 UTC m=+2.845337917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls") pod "image-registry-78749f569b-zzhxm" (UID: "7484d61b-5158-4729-b835-c0b372d8105b") : secret "image-registry-tls" not found Nov 02 17:29:43.224631 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.224146 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle podName:3d83bf78-e7b7-48bb-98be-8196e92fd320 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:44.224126238 +0000 UTC m=+2.845486005 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle") pod "router-default-79c77b5cd5-c6svb" (UID: "3d83bf78-e7b7-48bb-98be-8196e92fd320") : configmap references non-existent config key: service-ca.crt Nov 02 17:29:43.224631 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.224174 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls podName:a453427b-5025-4653-a930-ce2cf2f41cce nodeName:}" failed. No retries permitted until 2026-11-02 17:29:44.224162972 +0000 UTC m=+2.845522742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls") pod "dns-default-w7zsl" (UID: "a453427b-5025-4653-a930-ce2cf2f41cce") : secret "dns-default-metrics-tls" not found Nov 02 17:29:43.224631 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.224196 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls podName:85cbfa4a-71cd-4b2c-8596-a484deeb15a5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:44.224187 +0000 UTC m=+2.845546756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fbrw7" (UID: "85cbfa4a-71cd-4b2c-8596-a484deeb15a5") : secret "insights-runtime-extractor-tls" not found Nov 02 17:29:43.224631 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.224218 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls podName:42f9e736-411e-46a2-89d0-2a3870e2f95a nodeName:}" failed. No retries permitted until 2026-11-02 17:29:44.224209044 +0000 UTC m=+2.845568804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pthgk" (UID: "42f9e736-411e-46a2-89d0-2a3870e2f95a") : secret "cluster-monitoring-operator-tls" not found Nov 02 17:29:43.224631 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.224240 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs podName:3d83bf78-e7b7-48bb-98be-8196e92fd320 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:44.224232167 +0000 UTC m=+2.845591928 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs") pod "router-default-79c77b5cd5-c6svb" (UID: "3d83bf78-e7b7-48bb-98be-8196e92fd320") : secret "router-metrics-certs-default" not found Nov 02 17:29:43.224631 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:43.224303 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7scpf\" (UID: \"72445bf5-a7a3-49dc-9b97-5589ae901032\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:43.224631 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:43.224338 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:43.224631 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.224416 4860 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 17:29:43.224631 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.224458 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert podName:72445bf5-a7a3-49dc-9b97-5589ae901032 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:44.224447055 +0000 UTC m=+2.845806822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7scpf" (UID: "72445bf5-a7a3-49dc-9b97-5589ae901032") : secret "networking-console-plugin-cert" not found Nov 02 17:29:43.224631 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.224461 4860 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 17:29:43.224631 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.224477 4860 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8ffbdd9-xqf4t: secret "image-registry-tls" not found Nov 02 17:29:43.224631 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.224512 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls podName:27ea270b-6e55-4652-a1bf-1431a21c3d55 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:44.224500493 +0000 UTC m=+2.845860262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls") pod "image-registry-6f8ffbdd9-xqf4t" (UID: "27ea270b-6e55-4652-a1bf-1431a21c3d55") : secret "image-registry-tls" not found Nov 02 17:29:43.224631 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:43.224569 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:43.556102 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:43.556078 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 17:29:43.556488 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:43.556474 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/1.log" Nov 02 17:29:43.556523 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:43.556505 4860 generic.go:356] "Generic (PLEG): container finished" podID="5f9e8821-40a8-4012-b66a-813e80d856f2" containerID="9cdee7989c01683ddac730773174627cb4baed4400087f105aba7aca2ab17ff0" exitCode=255 Nov 02 17:29:43.556569 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:43.556549 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" event={"ID":"5f9e8821-40a8-4012-b66a-813e80d856f2","Type":"ContainerDied","Data":"9cdee7989c01683ddac730773174627cb4baed4400087f105aba7aca2ab17ff0"} Nov 02 17:29:43.556612 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:43.556600 4860 scope.go:122] "RemoveContainer" containerID="3d9ba56bcbd5f834961a29159d8acb53f61960759a915226ffb42f05b8a1ea33" Nov 02 17:29:43.557210 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:43.557190 4860 scope.go:122] "RemoveContainer" containerID="9cdee7989c01683ddac730773174627cb4baed4400087f105aba7aca2ab17ff0" Nov 02 17:29:43.557394 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:43.557376 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-xqzqj_openshift-console-operator(5f9e8821-40a8-4012-b66a-813e80d856f2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" podUID="5f9e8821-40a8-4012-b66a-813e80d856f2" Nov 02 17:29:44.236506 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:44.236470 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:44.236506 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:44.236512 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:44.236749 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:44.236538 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:44.236749 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.236624 4860 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 17:29:44.236749 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.236640 4860 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 17:29:44.236749 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.236652 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle podName:3d83bf78-e7b7-48bb-98be-8196e92fd320 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.236640259 +0000 UTC m=+4.858000026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle") pod "router-default-79c77b5cd5-c6svb" (UID: "3d83bf78-e7b7-48bb-98be-8196e92fd320") : configmap references non-existent config key: service-ca.crt Nov 02 17:29:44.236749 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:44.236695 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7scpf\" (UID: \"72445bf5-a7a3-49dc-9b97-5589ae901032\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:44.236749 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:44.236725 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:44.236749 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.236736 4860 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 17:29:44.237071 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:44.236764 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:44.237071 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.236776 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert podName:72445bf5-a7a3-49dc-9b97-5589ae901032 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.236760802 +0000 UTC m=+4.858120561 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7scpf" (UID: "72445bf5-a7a3-49dc-9b97-5589ae901032") : secret "networking-console-plugin-cert" not found Nov 02 17:29:44.237071 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.236803 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs podName:4b5c29e0-6bfd-4f43-867a-c460dc227fbc nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.2367927 +0000 UTC m=+4.858152457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs") pod "network-metrics-daemon-g6b2c" (UID: "4b5c29e0-6bfd-4f43-867a-c460dc227fbc") : secret "metrics-daemon-secret" not found Nov 02 17:29:44.237071 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.236807 4860 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 17:29:44.237071 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.236822 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls podName:a453427b-5025-4653-a930-ce2cf2f41cce nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.23681418 +0000 UTC m=+4.858173938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls") pod "dns-default-w7zsl" (UID: "a453427b-5025-4653-a930-ce2cf2f41cce") : secret "dns-default-metrics-tls" not found Nov 02 17:29:44.237071 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.236831 4860 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 17:29:44.237071 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.236843 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls podName:f20b07b4-cd27-4410-abb2-c0bd50c01269 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.236832507 +0000 UTC m=+4.858192260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dxkqn" (UID: "f20b07b4-cd27-4410-abb2-c0bd50c01269") : secret "samples-operator-tls" not found Nov 02 17:29:44.237071 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.236872 4860 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8ffbdd9-xqf4t: secret "image-registry-tls" not found Nov 02 17:29:44.237071 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:44.236886 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:44.237071 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.236908 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls podName:27ea270b-6e55-4652-a1bf-1431a21c3d55 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.236897282 +0000 UTC m=+4.858257036 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls") pod "image-registry-6f8ffbdd9-xqf4t" (UID: "27ea270b-6e55-4652-a1bf-1431a21c3d55") : secret "image-registry-tls" not found Nov 02 17:29:44.237071 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:44.236929 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:44.237071 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:44.236957 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:44.237071 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:44.236983 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:44.237071 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.236959 4860 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 17:29:44.237071 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.237037 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls podName:85cbfa4a-71cd-4b2c-8596-a484deeb15a5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.237027761 +0000 UTC m=+4.858387540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fbrw7" (UID: "85cbfa4a-71cd-4b2c-8596-a484deeb15a5") : secret "insights-runtime-extractor-tls" not found Nov 02 17:29:44.237071 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.237040 4860 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 17:29:44.237071 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.237001 4860 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 17:29:44.237573 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.237083 4860 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 17:29:44.237573 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.237107 4860 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78749f569b-zzhxm: secret "image-registry-tls" not found Nov 02 17:29:44.237573 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.237083 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs podName:3d83bf78-e7b7-48bb-98be-8196e92fd320 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.237071244 +0000 UTC m=+4.858431018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs") pod "router-default-79c77b5cd5-c6svb" (UID: "3d83bf78-e7b7-48bb-98be-8196e92fd320") : secret "router-metrics-certs-default" not found Nov 02 17:29:44.237573 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.237133 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls podName:42f9e736-411e-46a2-89d0-2a3870e2f95a nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.237120618 +0000 UTC m=+4.858480371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pthgk" (UID: "42f9e736-411e-46a2-89d0-2a3870e2f95a") : secret "cluster-monitoring-operator-tls" not found Nov 02 17:29:44.237573 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:44.237149 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls podName:7484d61b-5158-4729-b835-c0b372d8105b nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.237140771 +0000 UTC m=+4.858500544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls") pod "image-registry-78749f569b-zzhxm" (UID: "7484d61b-5158-4729-b835-c0b372d8105b") : secret "image-registry-tls" not found Nov 02 17:29:44.556487 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:44.556464 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s6s58_9368ef45-b865-4cdf-95a6-5088fc54277e/dns-node-resolver/0.log" Nov 02 17:29:44.560432 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:44.560412 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 17:29:44.689700 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:44.689665 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:29:45.354426 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:45.354400 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jxsgc_b44f96b4-5c43-40cc-9c89-63d4ec368e8b/node-ca/0.log" Nov 02 17:29:45.961732 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:45.961697 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4qfkf_704c917e-c41f-47c3-9a1e-02730361fc8a/serve-healthcheck-canary/0.log" Nov 02 17:29:46.256396 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.256310 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:46.256396 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.256350 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:46.256396 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.256386 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:46.256677 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.256406 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:46.256677 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.256433 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:46.256677 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.256478 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7scpf\" (UID: \"72445bf5-a7a3-49dc-9b97-5589ae901032\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:46.256677 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:46.256492 4860 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 17:29:46.256677 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.256512 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:46.256677 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:46.256573 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs podName:4b5c29e0-6bfd-4f43-867a-c460dc227fbc nodeName:}" failed. No retries permitted until 2026-11-02 17:29:50.256553997 +0000 UTC m=+8.877913765 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs") pod "network-metrics-daemon-g6b2c" (UID: "4b5c29e0-6bfd-4f43-867a-c460dc227fbc") : secret "metrics-daemon-secret" not found Nov 02 17:29:46.256677 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:46.256622 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle podName:3d83bf78-e7b7-48bb-98be-8196e92fd320 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:50.256599307 +0000 UTC m=+8.877959075 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle") pod "router-default-79c77b5cd5-c6svb" (UID: "3d83bf78-e7b7-48bb-98be-8196e92fd320") : configmap references non-existent config key: service-ca.crt Nov 02 17:29:46.256677 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:46.256636 4860 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 17:29:46.256677 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.256658 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:46.257195 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:46.256697 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert podName:72445bf5-a7a3-49dc-9b97-5589ae901032 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:50.256677895 +0000 UTC m=+8.878037649 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7scpf" (UID: "72445bf5-a7a3-49dc-9b97-5589ae901032") : secret "networking-console-plugin-cert" not found Nov 02 17:29:46.257195 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.256726 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:46.257195 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.256764 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:46.257195 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:46.256733 4860 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 17:29:46.257195 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.256730 4860 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 17:29:46.257195 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:46.256814 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls podName:f20b07b4-cd27-4410-abb2-c0bd50c01269 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:50.256804832 +0000 UTC m=+8.878164622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dxkqn" (UID: "f20b07b4-cd27-4410-abb2-c0bd50c01269") : secret "samples-operator-tls" not found Nov 02 17:29:46.257195 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:46.256775 4860 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 17:29:46.257195 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:46.256843 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls podName:85cbfa4a-71cd-4b2c-8596-a484deeb15a5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:50.256836327 +0000 UTC m=+8.878196121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fbrw7" (UID: "85cbfa4a-71cd-4b2c-8596-a484deeb15a5") : secret "insights-runtime-extractor-tls" not found Nov 02 17:29:46.259145 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.259118 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d83bf78-e7b7-48bb-98be-8196e92fd320-metrics-certs\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:46.259264 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.259150 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls\") pod \"image-registry-78749f569b-zzhxm\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:46.259264 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.259175 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42f9e736-411e-46a2-89d0-2a3870e2f95a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pthgk\" (UID: \"42f9e736-411e-46a2-89d0-2a3870e2f95a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:46.259401 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.259378 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a453427b-5025-4653-a930-ce2cf2f41cce-metrics-tls\") pod \"dns-default-w7zsl\" (UID: \"a453427b-5025-4653-a930-ce2cf2f41cce\") " pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:46.259472 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.259458 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls\") pod \"image-registry-6f8ffbdd9-xqf4t\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:46.399701 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.399672 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:46.401169 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.401142 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" Nov 02 17:29:46.409598 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.409512 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:46.411732 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.411650 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:46.484762 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.484662 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:46.485279 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.485152 4860 scope.go:122] "RemoveContainer" containerID="9cdee7989c01683ddac730773174627cb4baed4400087f105aba7aca2ab17ff0" Nov 02 17:29:46.485446 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:46.485393 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-xqzqj_openshift-console-operator(5f9e8821-40a8-4012-b66a-813e80d856f2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" podUID="5f9e8821-40a8-4012-b66a-813e80d856f2" Nov 02 17:29:46.730348 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.730314 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk"] Nov 02 17:29:46.733631 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:46.733596 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f9e736_411e_46a2_89d0_2a3870e2f95a.slice/crio-ba5c49050a45bb1dffa6f8fbfef5afa7c1a55a94b8efc82cfee5a3a17ef0d282 WatchSource:0}: Error finding container ba5c49050a45bb1dffa6f8fbfef5afa7c1a55a94b8efc82cfee5a3a17ef0d282: Status 404 returned error can't find the container with id ba5c49050a45bb1dffa6f8fbfef5afa7c1a55a94b8efc82cfee5a3a17ef0d282 Nov 02 17:29:46.734944 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.734921 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78749f569b-zzhxm"] Nov 02 17:29:46.735844 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.735828 4860 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:29:46.747204 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:46.747177 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7484d61b_5158_4729_b835_c0b372d8105b.slice/crio-71d4a71c53b24c8b39d5c8ff37cf047a221000c32774e0f36598eb84d5604156 WatchSource:0}: Error finding container 71d4a71c53b24c8b39d5c8ff37cf047a221000c32774e0f36598eb84d5604156: Status 404 returned error can't find the container with id 71d4a71c53b24c8b39d5c8ff37cf047a221000c32774e0f36598eb84d5604156 Nov 02 17:29:46.751661 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.751639 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f8ffbdd9-xqf4t"] Nov 02 17:29:46.756248 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:46.756225 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ea270b_6e55_4652_a1bf_1431a21c3d55.slice/crio-1cffb70c13331dd1dc4f0990999004896488895d92f3dfaab1f4c81b3158643b WatchSource:0}: Error finding container 1cffb70c13331dd1dc4f0990999004896488895d92f3dfaab1f4c81b3158643b: Status 404 returned error can't find the container with id 1cffb70c13331dd1dc4f0990999004896488895d92f3dfaab1f4c81b3158643b Nov 02 17:29:46.771427 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.771403 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w7zsl"] Nov 02 17:29:46.777136 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:46.777110 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda453427b_5025_4653_a930_ce2cf2f41cce.slice/crio-2cad4132cbec196780bb9a8b78b8a68058e76786a919b7307e8e2023677c5608 WatchSource:0}: Error finding container 2cad4132cbec196780bb9a8b78b8a68058e76786a919b7307e8e2023677c5608: Status 404 returned error can't find the container with id 2cad4132cbec196780bb9a8b78b8a68058e76786a919b7307e8e2023677c5608 Nov 02 17:29:46.954272 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:46.954241 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr_a074a9ac-624c-4828-9785-c8f2aecb5827/kube-storage-version-migrator-operator/0.log" Nov 02 17:29:47.574815 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:47.574771 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78749f569b-zzhxm" event={"ID":"7484d61b-5158-4729-b835-c0b372d8105b","Type":"ContainerStarted","Data":"d5eb12c5214189cc9bbba97336f1580f75bd6f29cd499c619a9554916f6e4b9b"} Nov 02 17:29:47.575321 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:47.574890 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78749f569b-zzhxm" event={"ID":"7484d61b-5158-4729-b835-c0b372d8105b","Type":"ContainerStarted","Data":"71d4a71c53b24c8b39d5c8ff37cf047a221000c32774e0f36598eb84d5604156"} Nov 02 17:29:47.575500 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:47.575478 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:29:47.577345 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:47.577319 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w7zsl" event={"ID":"a453427b-5025-4653-a930-ce2cf2f41cce","Type":"ContainerStarted","Data":"2cad4132cbec196780bb9a8b78b8a68058e76786a919b7307e8e2023677c5608"} Nov 02 17:29:47.579579 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:47.579245 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" event={"ID":"27ea270b-6e55-4652-a1bf-1431a21c3d55","Type":"ContainerStarted","Data":"4f98e4892c6ca1f946221d8800d0948c29dc17957d606a5b0e932d823e0115b1"} Nov 02 17:29:47.579579 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:47.579289 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" event={"ID":"27ea270b-6e55-4652-a1bf-1431a21c3d55","Type":"ContainerStarted","Data":"1cffb70c13331dd1dc4f0990999004896488895d92f3dfaab1f4c81b3158643b"} Nov 02 17:29:47.581056 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:47.581027 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" event={"ID":"42f9e736-411e-46a2-89d0-2a3870e2f95a","Type":"ContainerStarted","Data":"ba5c49050a45bb1dffa6f8fbfef5afa7c1a55a94b8efc82cfee5a3a17ef0d282"} Nov 02 17:29:48.304705 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:48.304675 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:48.589288 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:48.588464 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pthgk" event={"ID":"42f9e736-411e-46a2-89d0-2a3870e2f95a","Type":"ContainerStarted","Data":"2b812364375aa5260c493516388da5bc512b04921b4c390b135aa22eab66c68c"} Nov 02 17:29:48.591379 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:48.591357 4860 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:29:48.592377 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:48.591872 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w7zsl" event={"ID":"a453427b-5025-4653-a930-ce2cf2f41cce","Type":"ContainerStarted","Data":"6464fab52a07632858aeab51cc4b72c3408db71d3589314e803756305a82be1b"} Nov 02 17:29:48.665390 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:48.665355 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:29:48.665575 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:48.665563 4860 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:29:49.595896 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:49.595843 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w7zsl" event={"ID":"a453427b-5025-4653-a930-ce2cf2f41cce","Type":"ContainerStarted","Data":"b2421716a98e4de3fbb0a092c2308569cb523bb2ee9b4856dd0437c61134ac9c"} Nov 02 17:29:50.300556 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:50.300520 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:50.300764 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:50.300570 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7scpf\" (UID: \"72445bf5-a7a3-49dc-9b97-5589ae901032\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:50.300764 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:50.300603 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:50.300764 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:50.300622 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:50.300764 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:50.300659 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:50.300764 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:50.300699 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle podName:3d83bf78-e7b7-48bb-98be-8196e92fd320 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:58.300676301 +0000 UTC m=+16.922036069 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle") pod "router-default-79c77b5cd5-c6svb" (UID: "3d83bf78-e7b7-48bb-98be-8196e92fd320") : configmap references non-existent config key: service-ca.crt Nov 02 17:29:50.300764 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:50.300735 4860 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 17:29:50.300764 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:50.300745 4860 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 17:29:50.301037 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:50.300796 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert podName:72445bf5-a7a3-49dc-9b97-5589ae901032 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:58.300781252 +0000 UTC m=+16.922141021 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7scpf" (UID: "72445bf5-a7a3-49dc-9b97-5589ae901032") : secret "networking-console-plugin-cert" not found Nov 02 17:29:50.301037 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:50.300735 4860 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 17:29:50.301037 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:50.300815 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls podName:85cbfa4a-71cd-4b2c-8596-a484deeb15a5 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:58.300805856 +0000 UTC m=+16.922165614 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fbrw7" (UID: "85cbfa4a-71cd-4b2c-8596-a484deeb15a5") : secret "insights-runtime-extractor-tls" not found Nov 02 17:29:50.301037 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:50.300867 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls podName:f20b07b4-cd27-4410-abb2-c0bd50c01269 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:58.300832789 +0000 UTC m=+16.922192549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dxkqn" (UID: "f20b07b4-cd27-4410-abb2-c0bd50c01269") : secret "samples-operator-tls" not found Nov 02 17:29:50.302922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:50.302898 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5c29e0-6bfd-4f43-867a-c460dc227fbc-metrics-certs\") pod \"network-metrics-daemon-g6b2c\" (UID: \"4b5c29e0-6bfd-4f43-867a-c460dc227fbc\") " pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:50.310857 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:50.310827 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6b2c" Nov 02 17:29:50.599734 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:50.599699 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w7zsl" Nov 02 17:29:50.605679 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:50.605654 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g6b2c"] Nov 02 17:29:50.609347 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:50.609324 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b5c29e0_6bfd_4f43_867a_c460dc227fbc.slice/crio-698a4d00c5d4cf0e31495868d7382d765174af779e5b8349fd70f66d7c8072ad WatchSource:0}: Error finding container 698a4d00c5d4cf0e31495868d7382d765174af779e5b8349fd70f66d7c8072ad: Status 404 returned error can't find the container with id 698a4d00c5d4cf0e31495868d7382d765174af779e5b8349fd70f66d7c8072ad Nov 02 17:29:51.604091 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:51.604055 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g6b2c" event={"ID":"4b5c29e0-6bfd-4f43-867a-c460dc227fbc","Type":"ContainerStarted","Data":"55ea75f76746100a4b9dc7b68a3f1b3c22e2e375f60af17442bffde11c070e5c"} Nov 02 17:29:51.604499 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:51.604116 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g6b2c" event={"ID":"4b5c29e0-6bfd-4f43-867a-c460dc227fbc","Type":"ContainerStarted","Data":"3c65cf3c6af6f420917821294f4686c9d4e8eb8fef38a92ef4867edb338be4f8"} Nov 02 17:29:51.604499 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:51.604139 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g6b2c" event={"ID":"4b5c29e0-6bfd-4f43-867a-c460dc227fbc","Type":"ContainerStarted","Data":"698a4d00c5d4cf0e31495868d7382d765174af779e5b8349fd70f66d7c8072ad"} Nov 02 17:29:51.713455 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:51.713382 4860 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:51.713735 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:51.713722 4860 scope.go:122] "RemoveContainer" containerID="9cdee7989c01683ddac730773174627cb4baed4400087f105aba7aca2ab17ff0" Nov 02 17:29:51.713936 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:29:51.713915 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-xqzqj_openshift-console-operator(5f9e8821-40a8-4012-b66a-813e80d856f2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" podUID="5f9e8821-40a8-4012-b66a-813e80d856f2" Nov 02 17:29:51.910042 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:51.910006 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:29:54.491421 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:54.491391 4860 scope.go:122] "RemoveContainer" containerID="9cdee7989c01683ddac730773174627cb4baed4400087f105aba7aca2ab17ff0" Nov 02 17:29:54.615123 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:54.615099 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 17:29:54.615293 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:54.615194 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" event={"ID":"5f9e8821-40a8-4012-b66a-813e80d856f2","Type":"ContainerStarted","Data":"062cd2a8a2491334c54b9578b49fefbf1bb7d7e408c9cc246f7f213f9bb3755c"} Nov 02 17:29:54.615479 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:54.615459 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:54.616381 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:54.616356 4860 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-xqzqj container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.18:8443/readyz\": dial tcp 10.132.0.18:8443: connect: connection refused" start-of-body= Nov 02 17:29:54.616482 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:54.616408 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" podUID="5f9e8821-40a8-4012-b66a-813e80d856f2" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/readyz\": dial tcp 10.132.0.18:8443: connect: connection refused" Nov 02 17:29:55.622748 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:55.622717 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-xqzqj" Nov 02 17:29:58.095020 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:58.094985 4860 patch_prober.go:29] interesting pod/image-registry-78749f569b-zzhxm container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 17:29:58.095385 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:58.095045 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-78749f569b-zzhxm" podUID="7484d61b-5158-4729-b835-c0b372d8105b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:29:58.370774 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:58.370697 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:58.370774 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:58.370733 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:58.370774 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:58.370773 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:58.371025 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:58.370806 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7scpf\" (UID: \"72445bf5-a7a3-49dc-9b97-5589ae901032\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:58.371501 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:58.371470 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d83bf78-e7b7-48bb-98be-8196e92fd320-service-ca-bundle\") pod \"router-default-79c77b5cd5-c6svb\" (UID: \"3d83bf78-e7b7-48bb-98be-8196e92fd320\") " pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:58.373077 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:58.373044 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72445bf5-a7a3-49dc-9b97-5589ae901032-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7scpf\" (UID: \"72445bf5-a7a3-49dc-9b97-5589ae901032\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:58.373077 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:58.373070 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85cbfa4a-71cd-4b2c-8596-a484deeb15a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fbrw7\" (UID: \"85cbfa4a-71cd-4b2c-8596-a484deeb15a5\") " pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:58.373212 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:58.373078 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f20b07b4-cd27-4410-abb2-c0bd50c01269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dxkqn\" (UID: \"f20b07b4-cd27-4410-abb2-c0bd50c01269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:58.399986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:58.399964 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" Nov 02 17:29:58.400412 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:58.400396 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:29:58.402086 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:58.402071 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" Nov 02 17:29:58.412969 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:58.412910 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fbrw7" Nov 02 17:29:58.752934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:58.752906 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79c77b5cd5-c6svb"] Nov 02 17:29:58.754446 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:58.754421 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d83bf78_e7b7_48bb_98be_8196e92fd320.slice/crio-e0b744ff437c231218e686d8a401ecf87dee73e9a3a45c54e01f7588ba1f10c7 WatchSource:0}: Error finding container e0b744ff437c231218e686d8a401ecf87dee73e9a3a45c54e01f7588ba1f10c7: Status 404 returned error can't find the container with id e0b744ff437c231218e686d8a401ecf87dee73e9a3a45c54e01f7588ba1f10c7 Nov 02 17:29:58.756825 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:58.756796 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85cbfa4a_71cd_4b2c_8596_a484deeb15a5.slice/crio-21be788b551bae645564bb82e4cb1d832ae891aac1aeb7ce53511f93efbc6671 WatchSource:0}: Error finding container 21be788b551bae645564bb82e4cb1d832ae891aac1aeb7ce53511f93efbc6671: Status 404 returned error can't find the container with id 21be788b551bae645564bb82e4cb1d832ae891aac1aeb7ce53511f93efbc6671 Nov 02 17:29:58.765278 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:58.765254 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fbrw7"] Nov 02 17:29:58.766072 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:58.766055 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7scpf"] Nov 02 17:29:58.766651 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:58.766631 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn"] Nov 02 17:29:58.768748 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:29:58.768687 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72445bf5_a7a3_49dc_9b97_5589ae901032.slice/crio-c136d341fecbcb1fd0ca4e6cbd72318e8602cf981042545eb904d4b21f2d9459 WatchSource:0}: Error finding container c136d341fecbcb1fd0ca4e6cbd72318e8602cf981042545eb904d4b21f2d9459: Status 404 returned error can't find the container with id c136d341fecbcb1fd0ca4e6cbd72318e8602cf981042545eb904d4b21f2d9459 Nov 02 17:29:59.634869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:59.634707 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" event={"ID":"72445bf5-a7a3-49dc-9b97-5589ae901032","Type":"ContainerStarted","Data":"c136d341fecbcb1fd0ca4e6cbd72318e8602cf981042545eb904d4b21f2d9459"} Nov 02 17:29:59.636769 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:59.636737 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79c77b5cd5-c6svb" event={"ID":"3d83bf78-e7b7-48bb-98be-8196e92fd320","Type":"ContainerStarted","Data":"9c5ae0cbebf518eb08dc5e70e5963a3f911ba22f797a71f52c24cbfb2b812413"} Nov 02 17:29:59.636917 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:59.636786 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79c77b5cd5-c6svb" event={"ID":"3d83bf78-e7b7-48bb-98be-8196e92fd320","Type":"ContainerStarted","Data":"e0b744ff437c231218e686d8a401ecf87dee73e9a3a45c54e01f7588ba1f10c7"} Nov 02 17:29:59.638550 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:59.638518 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" event={"ID":"f20b07b4-cd27-4410-abb2-c0bd50c01269","Type":"ContainerStarted","Data":"5563da5410c4e8358882c4b76ab922ea66ec268122760aff4a9e13f6bcf083f8"} Nov 02 17:29:59.640809 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:59.640784 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fbrw7" event={"ID":"85cbfa4a-71cd-4b2c-8596-a484deeb15a5","Type":"ContainerStarted","Data":"6e35c1e5e7f1f7161356a98ff3d5e01132d7463cc00f4331bb9db411f8ec626f"} Nov 02 17:29:59.640928 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:59.640834 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fbrw7" event={"ID":"85cbfa4a-71cd-4b2c-8596-a484deeb15a5","Type":"ContainerStarted","Data":"c10609c45c8631490323337d62c6e419c19b171e9dee129a337489325ae5c1aa"} Nov 02 17:29:59.640928 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:29:59.640874 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fbrw7" event={"ID":"85cbfa4a-71cd-4b2c-8596-a484deeb15a5","Type":"ContainerStarted","Data":"21be788b551bae645564bb82e4cb1d832ae891aac1aeb7ce53511f93efbc6671"} Nov 02 17:30:00.400699 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:00.400662 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:30:00.403926 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:00.403898 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:30:00.606764 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:00.606733 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w7zsl" Nov 02 17:30:00.646634 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:00.646565 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" event={"ID":"f20b07b4-cd27-4410-abb2-c0bd50c01269","Type":"ContainerStarted","Data":"8f7abff4b705067a567a3f78c8f53bc436c672bf1177fbbfa7942dc01b0d3dc5"} Nov 02 17:30:00.646634 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:00.646624 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dxkqn" event={"ID":"f20b07b4-cd27-4410-abb2-c0bd50c01269","Type":"ContainerStarted","Data":"12c36ebf70c67b38586de31ba6054588ceeed65755b3a2ae860bbd774b1e1be0"} Nov 02 17:30:00.648072 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:00.648033 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-7scpf" event={"ID":"72445bf5-a7a3-49dc-9b97-5589ae901032","Type":"ContainerStarted","Data":"30f6550e7c085bcd15269be6d7d914e48bf5c8899b403661009107ac75d16e56"} Nov 02 17:30:00.648236 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:00.648216 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:30:00.649482 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:00.649461 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-79c77b5cd5-c6svb" Nov 02 17:30:01.395184 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:01.395149 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78749f569b-zzhxm"] Nov 02 17:30:01.399663 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:01.399632 4860 patch_prober.go:29] interesting pod/image-registry-78749f569b-zzhxm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 17:30:01.399807 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:01.399677 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-78749f569b-zzhxm" podUID="7484d61b-5158-4729-b835-c0b372d8105b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:30:01.654026 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:01.653897 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fbrw7" event={"ID":"85cbfa4a-71cd-4b2c-8596-a484deeb15a5","Type":"ContainerStarted","Data":"9d6b6d1f08d79269b8b81873f5da31049f6681e490e9f2f415cb0146127ae399"} Nov 02 17:30:01.869431 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:01.869395 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k58q5" Nov 02 17:30:01.873228 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:01.873201 4860 patch_prober.go:29] interesting pod/image-registry-6f8ffbdd9-xqf4t container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 17:30:01.873413 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:01.873240 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" podUID="27ea270b-6e55-4652-a1bf-1431a21c3d55" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:30:01.913999 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:01.913930 4860 patch_prober.go:29] interesting pod/image-registry-6f8ffbdd9-xqf4t container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 17:30:01.913999 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:01.913976 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" podUID="27ea270b-6e55-4652-a1bf-1431a21c3d55" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:30:06.454992 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:06.454958 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vs4r4" Nov 02 17:30:10.544721 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.544681 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xw25t"] Nov 02 17:30:10.580526 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.580495 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.671620 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.671585 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czqts\" (UniqueName: \"kubernetes.io/projected/729c02eb-e200-4928-b789-b312d38306e2-kube-api-access-czqts\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.671620 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.671619 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/729c02eb-e200-4928-b789-b312d38306e2-node-exporter-tls\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.671808 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.671654 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/729c02eb-e200-4928-b789-b312d38306e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.671808 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.671750 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/729c02eb-e200-4928-b789-b312d38306e2-node-exporter-wtmp\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.671808 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.671793 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/729c02eb-e200-4928-b789-b312d38306e2-root\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.671933 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.671828 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/729c02eb-e200-4928-b789-b312d38306e2-sys\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.671933 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.671868 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/729c02eb-e200-4928-b789-b312d38306e2-node-exporter-textfile\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.672027 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.671929 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/729c02eb-e200-4928-b789-b312d38306e2-metrics-client-ca\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.672062 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.672045 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/729c02eb-e200-4928-b789-b312d38306e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.679296 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.679276 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 17:30:10.679429 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.679414 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 17:30:10.683273 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.683250 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 17:30:10.683395 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.683331 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sn4p8\"" Nov 02 17:30:10.683807 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.683790 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 17:30:10.773030 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.772996 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/729c02eb-e200-4928-b789-b312d38306e2-root\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.773205 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.773041 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/729c02eb-e200-4928-b789-b312d38306e2-sys\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.773205 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.773062 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/729c02eb-e200-4928-b789-b312d38306e2-node-exporter-textfile\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.773205 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.773078 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/729c02eb-e200-4928-b789-b312d38306e2-metrics-client-ca\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.773205 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.773104 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/729c02eb-e200-4928-b789-b312d38306e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.773205 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.773132 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czqts\" (UniqueName: \"kubernetes.io/projected/729c02eb-e200-4928-b789-b312d38306e2-kube-api-access-czqts\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.773205 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.773125 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/729c02eb-e200-4928-b789-b312d38306e2-root\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.773205 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.773158 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/729c02eb-e200-4928-b789-b312d38306e2-node-exporter-tls\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.773205 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.773130 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/729c02eb-e200-4928-b789-b312d38306e2-sys\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.773582 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.773249 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/729c02eb-e200-4928-b789-b312d38306e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.773582 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.773291 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/729c02eb-e200-4928-b789-b312d38306e2-node-exporter-wtmp\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.773582 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.773452 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/729c02eb-e200-4928-b789-b312d38306e2-node-exporter-textfile\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.773582 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.773465 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/729c02eb-e200-4928-b789-b312d38306e2-node-exporter-wtmp\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.773784 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.773738 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/729c02eb-e200-4928-b789-b312d38306e2-metrics-client-ca\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.773838 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.773799 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/729c02eb-e200-4928-b789-b312d38306e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.775389 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.775365 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/729c02eb-e200-4928-b789-b312d38306e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.775585 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.775563 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/729c02eb-e200-4928-b789-b312d38306e2-node-exporter-tls\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.800395 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.800337 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czqts\" (UniqueName: \"kubernetes.io/projected/729c02eb-e200-4928-b789-b312d38306e2-kube-api-access-czqts\") pod \"node-exporter-xw25t\" (UID: \"729c02eb-e200-4928-b789-b312d38306e2\") " pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.932650 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:10.932619 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xw25t" Nov 02 17:30:10.968835 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:30:10.968794 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729c02eb_e200_4928_b789_b312d38306e2.slice/crio-d48afca55c7996a33eedf6e89efb1f1a3fb0b5bf417a559be42600ce02b3d095 WatchSource:0}: Error finding container d48afca55c7996a33eedf6e89efb1f1a3fb0b5bf417a559be42600ce02b3d095: Status 404 returned error can't find the container with id d48afca55c7996a33eedf6e89efb1f1a3fb0b5bf417a559be42600ce02b3d095 Nov 02 17:30:11.399315 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:11.399288 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:30:11.692385 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:11.692300 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xw25t" event={"ID":"729c02eb-e200-4928-b789-b312d38306e2","Type":"ContainerStarted","Data":"d48afca55c7996a33eedf6e89efb1f1a3fb0b5bf417a559be42600ce02b3d095"} Nov 02 17:30:11.915584 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:11.915554 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:30:12.697785 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:12.697706 4860 generic.go:356] "Generic (PLEG): container finished" podID="729c02eb-e200-4928-b789-b312d38306e2" containerID="dbf0893f1cbf3787f9831b4d8be066ad56d45d10ebc70470d50a2d8945200d43" exitCode=0 Nov 02 17:30:12.698205 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:12.697799 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xw25t" event={"ID":"729c02eb-e200-4928-b789-b312d38306e2","Type":"ContainerDied","Data":"dbf0893f1cbf3787f9831b4d8be066ad56d45d10ebc70470d50a2d8945200d43"} Nov 02 17:30:13.703625 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:13.703589 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xw25t" event={"ID":"729c02eb-e200-4928-b789-b312d38306e2","Type":"ContainerStarted","Data":"cfdfd3b41e99924f87b781b74462a8b82ce6684edb0b3aa83599a95292084e86"} Nov 02 17:30:13.704023 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:13.703655 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xw25t" event={"ID":"729c02eb-e200-4928-b789-b312d38306e2","Type":"ContainerStarted","Data":"e8248d6b1b7fb2859106801c4f82922928960ba3b46a5ea0ae78eea1b22961d3"} Nov 02 17:30:13.792917 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:13.792840 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xw25t" podStartSLOduration=2.289371063 podStartE2EDuration="3.79282443s" totalImagesPullingTime="1.423347529s" totalInitContainerRuntime="80.105838ms" isStatefulPod=true podCreationTimestamp="2026-11-02 17:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:13.778090404 +0000 UTC m=+32.399450180" watchObservedRunningTime="2026-11-02 17:30:13.79282443 +0000 UTC m=+32.414184205" Nov 02 17:30:14.692884 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:14.692833 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qjx25" Nov 02 17:30:21.768531 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:30:21.768494 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:30:24.829830 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:24.829748 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f8ffbdd9-xqf4t"] Nov 02 17:30:26.416451 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.416409 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-78749f569b-zzhxm" podUID="7484d61b-5158-4729-b835-c0b372d8105b" containerName="registry" containerID="cri-o://d5eb12c5214189cc9bbba97336f1580f75bd6f29cd499c619a9554916f6e4b9b" gracePeriod=30 Nov 02 17:30:26.660929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.660905 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:30:26.756792 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.756709 4860 generic.go:356] "Generic (PLEG): container finished" podID="7484d61b-5158-4729-b835-c0b372d8105b" containerID="d5eb12c5214189cc9bbba97336f1580f75bd6f29cd499c619a9554916f6e4b9b" exitCode=0 Nov 02 17:30:26.756792 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.756753 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78749f569b-zzhxm" Nov 02 17:30:26.756980 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.756792 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78749f569b-zzhxm" event={"ID":"7484d61b-5158-4729-b835-c0b372d8105b","Type":"ContainerDied","Data":"d5eb12c5214189cc9bbba97336f1580f75bd6f29cd499c619a9554916f6e4b9b"} Nov 02 17:30:26.756980 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.756838 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78749f569b-zzhxm" event={"ID":"7484d61b-5158-4729-b835-c0b372d8105b","Type":"ContainerDied","Data":"71d4a71c53b24c8b39d5c8ff37cf047a221000c32774e0f36598eb84d5604156"} Nov 02 17:30:26.756980 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.756887 4860 scope.go:122] "RemoveContainer" containerID="d5eb12c5214189cc9bbba97336f1580f75bd6f29cd499c619a9554916f6e4b9b" Nov 02 17:30:26.764592 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.764568 4860 scope.go:122] "RemoveContainer" containerID="d5eb12c5214189cc9bbba97336f1580f75bd6f29cd499c619a9554916f6e4b9b" Nov 02 17:30:26.764879 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:30:26.764835 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5eb12c5214189cc9bbba97336f1580f75bd6f29cd499c619a9554916f6e4b9b\": container with ID starting with d5eb12c5214189cc9bbba97336f1580f75bd6f29cd499c619a9554916f6e4b9b not found: ID does not exist" containerID="d5eb12c5214189cc9bbba97336f1580f75bd6f29cd499c619a9554916f6e4b9b" Nov 02 17:30:26.764946 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.764888 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5eb12c5214189cc9bbba97336f1580f75bd6f29cd499c619a9554916f6e4b9b"} err="failed to get container status \"d5eb12c5214189cc9bbba97336f1580f75bd6f29cd499c619a9554916f6e4b9b\": rpc error: code = NotFound desc = could not find container \"d5eb12c5214189cc9bbba97336f1580f75bd6f29cd499c619a9554916f6e4b9b\": container with ID starting with d5eb12c5214189cc9bbba97336f1580f75bd6f29cd499c619a9554916f6e4b9b not found: ID does not exist" Nov 02 17:30:26.814712 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.814683 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-installation-pull-secrets\") pod \"7484d61b-5158-4729-b835-c0b372d8105b\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " Nov 02 17:30:26.814836 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.814720 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-image-registry-private-configuration\") pod \"7484d61b-5158-4729-b835-c0b372d8105b\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " Nov 02 17:30:26.814836 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.814750 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-bound-sa-token\") pod \"7484d61b-5158-4729-b835-c0b372d8105b\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " Nov 02 17:30:26.814836 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.814789 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-trusted-ca\") pod \"7484d61b-5158-4729-b835-c0b372d8105b\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " Nov 02 17:30:26.815013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.814831 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-kube-api-access-2b4dt\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-kube-api-access-2b4dt\") pod \"7484d61b-5158-4729-b835-c0b372d8105b\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " Nov 02 17:30:26.815013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.814888 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7484d61b-5158-4729-b835-c0b372d8105b-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7484d61b-5158-4729-b835-c0b372d8105b-ca-trust-extracted\") pod \"7484d61b-5158-4729-b835-c0b372d8105b\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " Nov 02 17:30:26.815013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.814968 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls\") pod \"7484d61b-5158-4729-b835-c0b372d8105b\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " Nov 02 17:30:26.815170 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.815010 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-registry-certificates\") pod \"7484d61b-5158-4729-b835-c0b372d8105b\" (UID: \"7484d61b-5158-4729-b835-c0b372d8105b\") " Nov 02 17:30:26.815599 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.815501 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-trusted-ca" pod "7484d61b-5158-4729-b835-c0b372d8105b" (UID: "7484d61b-5158-4729-b835-c0b372d8105b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:30:26.815599 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.815539 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-registry-certificates" pod "7484d61b-5158-4729-b835-c0b372d8105b" (UID: "7484d61b-5158-4729-b835-c0b372d8105b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:30:26.817374 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.817314 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-image-registry-private-configuration" pod "7484d61b-5158-4729-b835-c0b372d8105b" (UID: "7484d61b-5158-4729-b835-c0b372d8105b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:30:26.817532 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.817478 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls" pod "7484d61b-5158-4729-b835-c0b372d8105b" (UID: "7484d61b-5158-4729-b835-c0b372d8105b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:30:26.817532 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.817524 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-kube-api-access-2b4dt" pod "7484d61b-5158-4729-b835-c0b372d8105b" (UID: "7484d61b-5158-4729-b835-c0b372d8105b"). InnerVolumeSpecName "kube-api-access-2b4dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:30:26.817721 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.817609 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-installation-pull-secrets" pod "7484d61b-5158-4729-b835-c0b372d8105b" (UID: "7484d61b-5158-4729-b835-c0b372d8105b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:30:26.817721 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.817645 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-bound-sa-token" pod "7484d61b-5158-4729-b835-c0b372d8105b" (UID: "7484d61b-5158-4729-b835-c0b372d8105b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:30:26.821601 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.821569 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7484d61b-5158-4729-b835-c0b372d8105b-ca-trust-extracted" pod "7484d61b-5158-4729-b835-c0b372d8105b" (UID: "7484d61b-5158-4729-b835-c0b372d8105b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:30:26.916278 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.916234 4860 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-installation-pull-secrets\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:30:26.916278 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.916268 4860 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7484d61b-5158-4729-b835-c0b372d8105b-image-registry-private-configuration\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:30:26.916278 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.916281 4860 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-bound-sa-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:30:26.916503 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.916290 4860 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-trusted-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:30:26.916503 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.916299 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b4dt\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-kube-api-access-2b4dt\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:30:26.916503 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.916307 4860 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7484d61b-5158-4729-b835-c0b372d8105b-ca-trust-extracted\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:30:26.916503 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.916316 4860 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484d61b-5158-4729-b835-c0b372d8105b-registry-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:30:26.916503 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:26.916324 4860 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7484d61b-5158-4729-b835-c0b372d8105b-registry-certificates\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:30:27.189011 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:27.188978 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78749f569b-zzhxm"] Nov 02 17:30:27.219954 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:27.219922 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-78749f569b-zzhxm"] Nov 02 17:30:27.497237 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:27.497152 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7484d61b-5158-4729-b835-c0b372d8105b" path="/var/lib/kubelet/pods/7484d61b-5158-4729-b835-c0b372d8105b/volumes" Nov 02 17:30:49.848682 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:49.848641 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" podUID="27ea270b-6e55-4652-a1bf-1431a21c3d55" containerName="registry" containerID="cri-o://4f98e4892c6ca1f946221d8800d0948c29dc17957d606a5b0e932d823e0115b1" gracePeriod=30 Nov 02 17:30:50.096569 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.096537 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:30:50.182346 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.182275 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-bound-sa-token\") pod \"27ea270b-6e55-4652-a1bf-1431a21c3d55\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " Nov 02 17:30:50.182346 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.182336 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-certificates\") pod \"27ea270b-6e55-4652-a1bf-1431a21c3d55\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " Nov 02 17:30:50.182562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.182356 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls\") pod \"27ea270b-6e55-4652-a1bf-1431a21c3d55\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " Nov 02 17:30:50.182562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.182398 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-installation-pull-secrets\") pod \"27ea270b-6e55-4652-a1bf-1431a21c3d55\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " Nov 02 17:30:50.182562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.182422 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-image-registry-private-configuration\") pod \"27ea270b-6e55-4652-a1bf-1431a21c3d55\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " Nov 02 17:30:50.182562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.182440 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ea270b-6e55-4652-a1bf-1431a21c3d55-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/27ea270b-6e55-4652-a1bf-1431a21c3d55-ca-trust-extracted\") pod \"27ea270b-6e55-4652-a1bf-1431a21c3d55\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " Nov 02 17:30:50.182562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.182487 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-trusted-ca\") pod \"27ea270b-6e55-4652-a1bf-1431a21c3d55\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " Nov 02 17:30:50.182562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.182516 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-kube-api-access-rpdh7\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-kube-api-access-rpdh7\") pod \"27ea270b-6e55-4652-a1bf-1431a21c3d55\" (UID: \"27ea270b-6e55-4652-a1bf-1431a21c3d55\") " Nov 02 17:30:50.182894 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.182731 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-certificates" pod "27ea270b-6e55-4652-a1bf-1431a21c3d55" (UID: "27ea270b-6e55-4652-a1bf-1431a21c3d55"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:30:50.183295 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.183264 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-trusted-ca" pod "27ea270b-6e55-4652-a1bf-1431a21c3d55" (UID: "27ea270b-6e55-4652-a1bf-1431a21c3d55"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:30:50.185008 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.184976 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-kube-api-access-rpdh7" pod "27ea270b-6e55-4652-a1bf-1431a21c3d55" (UID: "27ea270b-6e55-4652-a1bf-1431a21c3d55"). InnerVolumeSpecName "kube-api-access-rpdh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:30:50.185123 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.185053 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls" pod "27ea270b-6e55-4652-a1bf-1431a21c3d55" (UID: "27ea270b-6e55-4652-a1bf-1431a21c3d55"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:30:50.185311 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.185280 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-image-registry-private-configuration" pod "27ea270b-6e55-4652-a1bf-1431a21c3d55" (UID: "27ea270b-6e55-4652-a1bf-1431a21c3d55"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:30:50.185618 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.185590 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-installation-pull-secrets" pod "27ea270b-6e55-4652-a1bf-1431a21c3d55" (UID: "27ea270b-6e55-4652-a1bf-1431a21c3d55"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:30:50.185699 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.185683 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-bound-sa-token" pod "27ea270b-6e55-4652-a1bf-1431a21c3d55" (UID: "27ea270b-6e55-4652-a1bf-1431a21c3d55"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:30:50.192106 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.192072 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ea270b-6e55-4652-a1bf-1431a21c3d55-ca-trust-extracted" pod "27ea270b-6e55-4652-a1bf-1431a21c3d55" (UID: "27ea270b-6e55-4652-a1bf-1431a21c3d55"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:30:50.284037 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.284001 4860 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-image-registry-private-configuration\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:30:50.284037 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.284038 4860 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/27ea270b-6e55-4652-a1bf-1431a21c3d55-ca-trust-extracted\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:30:50.284275 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.284051 4860 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-trusted-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:30:50.284275 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.284060 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpdh7\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-kube-api-access-rpdh7\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:30:50.284275 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.284069 4860 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-bound-sa-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:30:50.284275 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.284078 4860 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-certificates\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:30:50.284275 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.284088 4860 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27ea270b-6e55-4652-a1bf-1431a21c3d55-registry-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:30:50.284275 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.284097 4860 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/27ea270b-6e55-4652-a1bf-1431a21c3d55-installation-pull-secrets\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:30:50.851919 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.851878 4860 generic.go:356] "Generic (PLEG): container finished" podID="27ea270b-6e55-4652-a1bf-1431a21c3d55" containerID="4f98e4892c6ca1f946221d8800d0948c29dc17957d606a5b0e932d823e0115b1" exitCode=0 Nov 02 17:30:50.851919 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.851919 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" Nov 02 17:30:50.852382 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.851931 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" event={"ID":"27ea270b-6e55-4652-a1bf-1431a21c3d55","Type":"ContainerDied","Data":"4f98e4892c6ca1f946221d8800d0948c29dc17957d606a5b0e932d823e0115b1"} Nov 02 17:30:50.852382 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.851984 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8ffbdd9-xqf4t" event={"ID":"27ea270b-6e55-4652-a1bf-1431a21c3d55","Type":"ContainerDied","Data":"1cffb70c13331dd1dc4f0990999004896488895d92f3dfaab1f4c81b3158643b"} Nov 02 17:30:50.852382 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.852011 4860 scope.go:122] "RemoveContainer" containerID="4f98e4892c6ca1f946221d8800d0948c29dc17957d606a5b0e932d823e0115b1" Nov 02 17:30:50.860447 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.860427 4860 scope.go:122] "RemoveContainer" containerID="4f98e4892c6ca1f946221d8800d0948c29dc17957d606a5b0e932d823e0115b1" Nov 02 17:30:50.860734 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:30:50.860713 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f98e4892c6ca1f946221d8800d0948c29dc17957d606a5b0e932d823e0115b1\": container with ID starting with 4f98e4892c6ca1f946221d8800d0948c29dc17957d606a5b0e932d823e0115b1 not found: ID does not exist" containerID="4f98e4892c6ca1f946221d8800d0948c29dc17957d606a5b0e932d823e0115b1" Nov 02 17:30:50.860780 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.860745 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f98e4892c6ca1f946221d8800d0948c29dc17957d606a5b0e932d823e0115b1"} err="failed to get container status \"4f98e4892c6ca1f946221d8800d0948c29dc17957d606a5b0e932d823e0115b1\": rpc error: code = NotFound desc = could not find container \"4f98e4892c6ca1f946221d8800d0948c29dc17957d606a5b0e932d823e0115b1\": container with ID starting with 4f98e4892c6ca1f946221d8800d0948c29dc17957d606a5b0e932d823e0115b1 not found: ID does not exist" Nov 02 17:30:50.924600 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.924566 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f8ffbdd9-xqf4t"] Nov 02 17:30:50.938618 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:50.938590 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f8ffbdd9-xqf4t"] Nov 02 17:30:51.496347 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:51.496318 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ea270b-6e55-4652-a1bf-1431a21c3d55" path="/var/lib/kubelet/pods/27ea270b-6e55-4652-a1bf-1431a21c3d55/volumes" Nov 02 17:30:51.768308 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:30:51.768226 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:30:57.878879 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:57.878828 4860 generic.go:356] "Generic (PLEG): container finished" podID="bc4a6db8-98e0-4387-8e64-1020978970c3" containerID="8662042c41316b6feddd4582a37f359ed6154b11db0c8fdc6eb54549fbd09633" exitCode=0 Nov 02 17:30:57.879302 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:57.878904 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" event={"ID":"bc4a6db8-98e0-4387-8e64-1020978970c3","Type":"ContainerDied","Data":"8662042c41316b6feddd4582a37f359ed6154b11db0c8fdc6eb54549fbd09633"} Nov 02 17:30:57.879302 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:57.879221 4860 scope.go:122] "RemoveContainer" containerID="8662042c41316b6feddd4582a37f359ed6154b11db0c8fdc6eb54549fbd09633" Nov 02 17:30:58.883810 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:30:58.883772 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zbfmn" event={"ID":"bc4a6db8-98e0-4387-8e64-1020978970c3","Type":"ContainerStarted","Data":"6457585b1a870b87c6ecfaa95585eb009dcc5158b6608bfe225e1d23bbeca39c"} Nov 02 17:31:02.903534 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:02.903499 4860 generic.go:356] "Generic (PLEG): container finished" podID="a074a9ac-624c-4828-9785-c8f2aecb5827" containerID="9d96d086ad1f95c5550472add0b63ad439e1e3b5ce53a6d24bdbba7e89db53ed" exitCode=0 Nov 02 17:31:02.903942 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:02.903580 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" event={"ID":"a074a9ac-624c-4828-9785-c8f2aecb5827","Type":"ContainerDied","Data":"9d96d086ad1f95c5550472add0b63ad439e1e3b5ce53a6d24bdbba7e89db53ed"} Nov 02 17:31:02.904024 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:02.904008 4860 scope.go:122] "RemoveContainer" containerID="9d96d086ad1f95c5550472add0b63ad439e1e3b5ce53a6d24bdbba7e89db53ed" Nov 02 17:31:03.909455 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:03.909417 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rhmwr" event={"ID":"a074a9ac-624c-4828-9785-c8f2aecb5827","Type":"ContainerStarted","Data":"685af72c408dc92afade57390c4680c9b3bf091aa627cf8c5eb23b51152ca961"} Nov 02 17:31:12.943329 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:12.943293 4860 generic.go:356] "Generic (PLEG): container finished" podID="7e4dc8ce-2802-4864-aed5-affeefe41d6b" containerID="cf64d6d85efbfa52c4ba9c95a29f449a4685ac8cf8f950a0e41dc88ea9913122" exitCode=0 Nov 02 17:31:12.943719 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:12.943368 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tszgp" event={"ID":"7e4dc8ce-2802-4864-aed5-affeefe41d6b","Type":"ContainerDied","Data":"cf64d6d85efbfa52c4ba9c95a29f449a4685ac8cf8f950a0e41dc88ea9913122"} Nov 02 17:31:12.943719 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:12.943696 4860 scope.go:122] "RemoveContainer" containerID="cf64d6d85efbfa52c4ba9c95a29f449a4685ac8cf8f950a0e41dc88ea9913122" Nov 02 17:31:13.556609 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.556570 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-86mkk-5hzgx"] Nov 02 17:31:13.556960 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.556943 4860 cpu_manager.go:414] "containerMap: removing container" podUID="7484d61b-5158-4729-b835-c0b372d8105b" containerName="registry" Nov 02 17:31:13.557063 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.556962 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7484d61b-5158-4729-b835-c0b372d8105b" containerName="registry" Nov 02 17:31:13.557063 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.556981 4860 cpu_manager.go:414] "containerMap: removing container" podUID="27ea270b-6e55-4652-a1bf-1431a21c3d55" containerName="registry" Nov 02 17:31:13.557063 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.556988 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ea270b-6e55-4652-a1bf-1431a21c3d55" containerName="registry" Nov 02 17:31:13.557209 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.557072 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="7484d61b-5158-4729-b835-c0b372d8105b" containerName="registry" Nov 02 17:31:13.557209 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.557084 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ea270b-6e55-4652-a1bf-1431a21c3d55" containerName="registry" Nov 02 17:31:13.560149 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.560128 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-86mkk-5hzgx" Nov 02 17:31:13.581833 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.581805 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-86mkk-5hzgx"] Nov 02 17:31:13.664195 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.664153 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vfkp\" (UniqueName: \"kubernetes.io/projected/0caba324-7d75-479c-a1ca-1af2ec3819cb-kube-api-access-5vfkp\") pod \"periodic-gathering-86mkk-5hzgx\" (UID: \"0caba324-7d75-479c-a1ca-1af2ec3819cb\") " pod="openshift-insights/periodic-gathering-86mkk-5hzgx" Nov 02 17:31:13.664365 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.664220 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0caba324-7d75-479c-a1ca-1af2ec3819cb-service-ca-bundle\") pod \"periodic-gathering-86mkk-5hzgx\" (UID: \"0caba324-7d75-479c-a1ca-1af2ec3819cb\") " pod="openshift-insights/periodic-gathering-86mkk-5hzgx" Nov 02 17:31:13.664365 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.664313 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0caba324-7d75-479c-a1ca-1af2ec3819cb-trusted-ca-bundle\") pod \"periodic-gathering-86mkk-5hzgx\" (UID: \"0caba324-7d75-479c-a1ca-1af2ec3819cb\") " pod="openshift-insights/periodic-gathering-86mkk-5hzgx" Nov 02 17:31:13.664365 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.664347 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0caba324-7d75-479c-a1ca-1af2ec3819cb-archives-path\") pod \"periodic-gathering-86mkk-5hzgx\" (UID: \"0caba324-7d75-479c-a1ca-1af2ec3819cb\") " pod="openshift-insights/periodic-gathering-86mkk-5hzgx" Nov 02 17:31:13.765681 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.765641 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0caba324-7d75-479c-a1ca-1af2ec3819cb-service-ca-bundle\") pod \"periodic-gathering-86mkk-5hzgx\" (UID: \"0caba324-7d75-479c-a1ca-1af2ec3819cb\") " pod="openshift-insights/periodic-gathering-86mkk-5hzgx" Nov 02 17:31:13.765839 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.765693 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0caba324-7d75-479c-a1ca-1af2ec3819cb-trusted-ca-bundle\") pod \"periodic-gathering-86mkk-5hzgx\" (UID: \"0caba324-7d75-479c-a1ca-1af2ec3819cb\") " pod="openshift-insights/periodic-gathering-86mkk-5hzgx" Nov 02 17:31:13.765839 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.765807 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0caba324-7d75-479c-a1ca-1af2ec3819cb-archives-path\") pod \"periodic-gathering-86mkk-5hzgx\" (UID: \"0caba324-7d75-479c-a1ca-1af2ec3819cb\") " pod="openshift-insights/periodic-gathering-86mkk-5hzgx" Nov 02 17:31:13.765945 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.765903 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vfkp\" (UniqueName: \"kubernetes.io/projected/0caba324-7d75-479c-a1ca-1af2ec3819cb-kube-api-access-5vfkp\") pod \"periodic-gathering-86mkk-5hzgx\" (UID: \"0caba324-7d75-479c-a1ca-1af2ec3819cb\") " pod="openshift-insights/periodic-gathering-86mkk-5hzgx" Nov 02 17:31:13.766206 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.766183 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0caba324-7d75-479c-a1ca-1af2ec3819cb-archives-path\") pod \"periodic-gathering-86mkk-5hzgx\" (UID: \"0caba324-7d75-479c-a1ca-1af2ec3819cb\") " pod="openshift-insights/periodic-gathering-86mkk-5hzgx" Nov 02 17:31:13.766354 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.766337 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0caba324-7d75-479c-a1ca-1af2ec3819cb-service-ca-bundle\") pod \"periodic-gathering-86mkk-5hzgx\" (UID: \"0caba324-7d75-479c-a1ca-1af2ec3819cb\") " pod="openshift-insights/periodic-gathering-86mkk-5hzgx" Nov 02 17:31:13.766473 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.766456 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0caba324-7d75-479c-a1ca-1af2ec3819cb-trusted-ca-bundle\") pod \"periodic-gathering-86mkk-5hzgx\" (UID: \"0caba324-7d75-479c-a1ca-1af2ec3819cb\") " pod="openshift-insights/periodic-gathering-86mkk-5hzgx" Nov 02 17:31:13.792130 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.792100 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vfkp\" (UniqueName: \"kubernetes.io/projected/0caba324-7d75-479c-a1ca-1af2ec3819cb-kube-api-access-5vfkp\") pod \"periodic-gathering-86mkk-5hzgx\" (UID: \"0caba324-7d75-479c-a1ca-1af2ec3819cb\") " pod="openshift-insights/periodic-gathering-86mkk-5hzgx" Nov 02 17:31:13.871367 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.871297 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-86mkk-5hzgx" Nov 02 17:31:13.949706 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:13.949668 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tszgp" event={"ID":"7e4dc8ce-2802-4864-aed5-affeefe41d6b","Type":"ContainerStarted","Data":"e442df3d9fd5adb6b6c37a88f354a3411ee55222da495f7c2352e6be6dfa758a"} Nov 02 17:31:14.163357 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:14.163275 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-86mkk-5hzgx"] Nov 02 17:31:14.166151 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:31:14.166115 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0caba324_7d75_479c_a1ca_1af2ec3819cb.slice/crio-ef0d2ef89d4bf6e9880f885caf8efd7b71ac06bab4d2f288cf594fd4b8141684 WatchSource:0}: Error finding container ef0d2ef89d4bf6e9880f885caf8efd7b71ac06bab4d2f288cf594fd4b8141684: Status 404 returned error can't find the container with id ef0d2ef89d4bf6e9880f885caf8efd7b71ac06bab4d2f288cf594fd4b8141684 Nov 02 17:31:14.954915 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:14.954881 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-86mkk-5hzgx" event={"ID":"0caba324-7d75-479c-a1ca-1af2ec3819cb","Type":"ContainerStarted","Data":"33ef38cf0610fe3b87018c2abfe80f0626a235f91b94c181b083337175f0020d"} Nov 02 17:31:14.955305 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:14.954935 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-86mkk-5hzgx" event={"ID":"0caba324-7d75-479c-a1ca-1af2ec3819cb","Type":"ContainerStarted","Data":"ef0d2ef89d4bf6e9880f885caf8efd7b71ac06bab4d2f288cf594fd4b8141684"} Nov 02 17:31:15.000660 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:31:15.000608 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-86mkk-5hzgx" podStartSLOduration=2.000592619 podStartE2EDuration="2.000592619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:31:14.999267145 +0000 UTC m=+93.620626921" watchObservedRunningTime="2026-11-02 17:31:15.000592619 +0000 UTC m=+93.621952396" Nov 02 17:31:21.768952 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:31:21.768917 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:31:51.769332 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:31:51.769294 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:32:21.768675 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:32:21.768592 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:32:26.299929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:26.299891 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7"] Nov 02 17:32:26.303450 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:26.303427 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7" Nov 02 17:32:26.320161 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:26.320139 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 17:32:26.331317 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:26.331293 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-fztfg\"" Nov 02 17:32:26.331450 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:26.331355 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 17:32:26.332383 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:26.332363 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7"] Nov 02 17:32:26.451205 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:26.451172 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/565e578b-d2dc-4fb3-987f-70d19da8148c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7\" (UID: \"565e578b-d2dc-4fb3-987f-70d19da8148c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7" Nov 02 17:32:26.451205 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:26.451213 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/565e578b-d2dc-4fb3-987f-70d19da8148c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7\" (UID: \"565e578b-d2dc-4fb3-987f-70d19da8148c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7" Nov 02 17:32:26.451470 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:26.451397 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rpqg\" (UniqueName: \"kubernetes.io/projected/565e578b-d2dc-4fb3-987f-70d19da8148c-kube-api-access-8rpqg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7\" (UID: \"565e578b-d2dc-4fb3-987f-70d19da8148c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7" Nov 02 17:32:26.552240 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:26.552134 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rpqg\" (UniqueName: \"kubernetes.io/projected/565e578b-d2dc-4fb3-987f-70d19da8148c-kube-api-access-8rpqg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7\" (UID: \"565e578b-d2dc-4fb3-987f-70d19da8148c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7" Nov 02 17:32:26.552240 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:26.552206 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/565e578b-d2dc-4fb3-987f-70d19da8148c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7\" (UID: \"565e578b-d2dc-4fb3-987f-70d19da8148c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7" Nov 02 17:32:26.552240 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:26.552241 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/565e578b-d2dc-4fb3-987f-70d19da8148c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7\" (UID: \"565e578b-d2dc-4fb3-987f-70d19da8148c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7" Nov 02 17:32:26.552667 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:26.552642 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/565e578b-d2dc-4fb3-987f-70d19da8148c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7\" (UID: \"565e578b-d2dc-4fb3-987f-70d19da8148c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7" Nov 02 17:32:26.552747 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:26.552684 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/565e578b-d2dc-4fb3-987f-70d19da8148c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7\" (UID: \"565e578b-d2dc-4fb3-987f-70d19da8148c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7" Nov 02 17:32:26.576015 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:26.575978 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rpqg\" (UniqueName: \"kubernetes.io/projected/565e578b-d2dc-4fb3-987f-70d19da8148c-kube-api-access-8rpqg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7\" (UID: \"565e578b-d2dc-4fb3-987f-70d19da8148c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7" Nov 02 17:32:26.619060 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:26.619024 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7" Nov 02 17:32:26.922478 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:26.922454 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7"] Nov 02 17:32:26.925164 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:32:26.925135 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565e578b_d2dc_4fb3_987f_70d19da8148c.slice/crio-00d910888423ac9b444d52701c283de2af4745c87e0be63f693e93859d2f7b09 WatchSource:0}: Error finding container 00d910888423ac9b444d52701c283de2af4745c87e0be63f693e93859d2f7b09: Status 404 returned error can't find the container with id 00d910888423ac9b444d52701c283de2af4745c87e0be63f693e93859d2f7b09 Nov 02 17:32:27.216128 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:27.216045 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7" event={"ID":"565e578b-d2dc-4fb3-987f-70d19da8148c","Type":"ContainerStarted","Data":"00d910888423ac9b444d52701c283de2af4745c87e0be63f693e93859d2f7b09"} Nov 02 17:32:27.887363 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:27.887327 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x"] Nov 02 17:32:27.890565 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:27.890544 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x" Nov 02 17:32:27.921289 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:27.921261 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-nzhtd\"" Nov 02 17:32:27.922322 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:27.922286 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x"] Nov 02 17:32:28.063661 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.063621 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x\" (UID: \"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x" Nov 02 17:32:28.063906 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.063677 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x\" (UID: \"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x" Nov 02 17:32:28.063906 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.063865 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c4mn\" (UniqueName: \"kubernetes.io/projected/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-kube-api-access-7c4mn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x\" (UID: \"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x" Nov 02 17:32:28.077168 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.077129 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q"] Nov 02 17:32:28.080454 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.080434 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q" Nov 02 17:32:28.095177 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.095155 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-9z79q\"" Nov 02 17:32:28.107424 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.107393 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q"] Nov 02 17:32:28.165066 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.164979 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x\" (UID: \"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x" Nov 02 17:32:28.165066 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.165023 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x\" (UID: \"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x" Nov 02 17:32:28.165066 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.165062 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c4mn\" (UniqueName: \"kubernetes.io/projected/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-kube-api-access-7c4mn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x\" (UID: \"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x" Nov 02 17:32:28.165471 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.165432 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x\" (UID: \"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x" Nov 02 17:32:28.165471 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.165459 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x\" (UID: \"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x" Nov 02 17:32:28.170779 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.170749 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd"] Nov 02 17:32:28.174392 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.174367 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd" Nov 02 17:32:28.197181 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.197156 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-wbs65\"" Nov 02 17:32:28.197658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.197632 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd"] Nov 02 17:32:28.204161 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.204132 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c4mn\" (UniqueName: \"kubernetes.io/projected/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-kube-api-access-7c4mn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x\" (UID: \"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x" Nov 02 17:32:28.206813 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.206790 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x" Nov 02 17:32:28.269602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.269562 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95psk\" (UniqueName: \"kubernetes.io/projected/3e11a3a4-96e3-42da-905a-4ccb26fde15e-kube-api-access-95psk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd\" (UID: \"3e11a3a4-96e3-42da-905a-4ccb26fde15e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd" Nov 02 17:32:28.269715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.269627 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e11a3a4-96e3-42da-905a-4ccb26fde15e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd\" (UID: \"3e11a3a4-96e3-42da-905a-4ccb26fde15e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd" Nov 02 17:32:28.269715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.269664 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrsh\" (UniqueName: \"kubernetes.io/projected/c02198c6-d4e4-4bda-90e6-182f01fa068a-kube-api-access-mhrsh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q\" (UID: \"c02198c6-d4e4-4bda-90e6-182f01fa068a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q" Nov 02 17:32:28.269715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.269703 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e11a3a4-96e3-42da-905a-4ccb26fde15e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd\" (UID: \"3e11a3a4-96e3-42da-905a-4ccb26fde15e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd" Nov 02 17:32:28.270676 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.270645 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4"] Nov 02 17:32:28.276016 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.275989 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c02198c6-d4e4-4bda-90e6-182f01fa068a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q\" (UID: \"c02198c6-d4e4-4bda-90e6-182f01fa068a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q" Nov 02 17:32:28.276103 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.276052 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c02198c6-d4e4-4bda-90e6-182f01fa068a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q\" (UID: \"c02198c6-d4e4-4bda-90e6-182f01fa068a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q" Nov 02 17:32:28.276168 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.276146 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" Nov 02 17:32:28.296941 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.296913 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4"] Nov 02 17:32:28.297364 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.297328 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-tknwf\"" Nov 02 17:32:28.368323 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.368286 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9"] Nov 02 17:32:28.371997 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.371969 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9" Nov 02 17:32:28.376813 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.376785 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c02198c6-d4e4-4bda-90e6-182f01fa068a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q\" (UID: \"c02198c6-d4e4-4bda-90e6-182f01fa068a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q" Nov 02 17:32:28.376966 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.376844 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c02198c6-d4e4-4bda-90e6-182f01fa068a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q\" (UID: \"c02198c6-d4e4-4bda-90e6-182f01fa068a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q" Nov 02 17:32:28.376966 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.376925 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95psk\" (UniqueName: \"kubernetes.io/projected/3e11a3a4-96e3-42da-905a-4ccb26fde15e-kube-api-access-95psk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd\" (UID: \"3e11a3a4-96e3-42da-905a-4ccb26fde15e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd" Nov 02 17:32:28.377120 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.377094 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e11a3a4-96e3-42da-905a-4ccb26fde15e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd\" (UID: \"3e11a3a4-96e3-42da-905a-4ccb26fde15e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd" Nov 02 17:32:28.377230 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.377212 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c02198c6-d4e4-4bda-90e6-182f01fa068a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q\" (UID: \"c02198c6-d4e4-4bda-90e6-182f01fa068a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q" Nov 02 17:32:28.377304 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.377238 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrsh\" (UniqueName: \"kubernetes.io/projected/c02198c6-d4e4-4bda-90e6-182f01fa068a-kube-api-access-mhrsh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q\" (UID: \"c02198c6-d4e4-4bda-90e6-182f01fa068a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q" Nov 02 17:32:28.377304 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.377267 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e11a3a4-96e3-42da-905a-4ccb26fde15e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd\" (UID: \"3e11a3a4-96e3-42da-905a-4ccb26fde15e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd" Nov 02 17:32:28.377304 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.377293 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c02198c6-d4e4-4bda-90e6-182f01fa068a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q\" (UID: \"c02198c6-d4e4-4bda-90e6-182f01fa068a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q" Nov 02 17:32:28.377490 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.377469 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e11a3a4-96e3-42da-905a-4ccb26fde15e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd\" (UID: \"3e11a3a4-96e3-42da-905a-4ccb26fde15e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd" Nov 02 17:32:28.377567 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.377547 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e11a3a4-96e3-42da-905a-4ccb26fde15e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd\" (UID: \"3e11a3a4-96e3-42da-905a-4ccb26fde15e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd" Nov 02 17:32:28.392013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.391986 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9"] Nov 02 17:32:28.406601 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.406579 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-6ztdj\"" Nov 02 17:32:28.409700 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.409673 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrsh\" (UniqueName: \"kubernetes.io/projected/c02198c6-d4e4-4bda-90e6-182f01fa068a-kube-api-access-mhrsh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q\" (UID: \"c02198c6-d4e4-4bda-90e6-182f01fa068a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q" Nov 02 17:32:28.409799 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.409778 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95psk\" (UniqueName: \"kubernetes.io/projected/3e11a3a4-96e3-42da-905a-4ccb26fde15e-kube-api-access-95psk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd\" (UID: \"3e11a3a4-96e3-42da-905a-4ccb26fde15e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd" Nov 02 17:32:28.478072 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.477987 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d32a6246-abfe-40fa-921d-560e9443aeb0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9\" (UID: \"d32a6246-abfe-40fa-921d-560e9443aeb0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9" Nov 02 17:32:28.478228 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.478092 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4\" (UID: \"d50aed5b-c7b1-4ac2-bb46-d5013752af0e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" Nov 02 17:32:28.478228 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.478136 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr9k9\" (UniqueName: \"kubernetes.io/projected/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-kube-api-access-kr9k9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4\" (UID: \"d50aed5b-c7b1-4ac2-bb46-d5013752af0e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" Nov 02 17:32:28.478228 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.478170 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d32a6246-abfe-40fa-921d-560e9443aeb0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9\" (UID: \"d32a6246-abfe-40fa-921d-560e9443aeb0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9" Nov 02 17:32:28.478355 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.478275 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zrz5\" (UniqueName: \"kubernetes.io/projected/d32a6246-abfe-40fa-921d-560e9443aeb0-kube-api-access-9zrz5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9\" (UID: \"d32a6246-abfe-40fa-921d-560e9443aeb0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9" Nov 02 17:32:28.478355 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.478342 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4\" (UID: \"d50aed5b-c7b1-4ac2-bb46-d5013752af0e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" Nov 02 17:32:28.492911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.492879 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd" Nov 02 17:32:28.514400 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.514370 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x"] Nov 02 17:32:28.519086 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:32:28.519058 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod062ebaf9_1d88_41af_9d5e_5b233f2a1eeb.slice/crio-6d781d50814c287f41988e94a64cdf7ce95998661ff62104d8a5164df6a00480 WatchSource:0}: Error finding container 6d781d50814c287f41988e94a64cdf7ce95998661ff62104d8a5164df6a00480: Status 404 returned error can't find the container with id 6d781d50814c287f41988e94a64cdf7ce95998661ff62104d8a5164df6a00480 Nov 02 17:32:28.579358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.579318 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d32a6246-abfe-40fa-921d-560e9443aeb0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9\" (UID: \"d32a6246-abfe-40fa-921d-560e9443aeb0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9" Nov 02 17:32:28.579517 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.579381 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4\" (UID: \"d50aed5b-c7b1-4ac2-bb46-d5013752af0e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" Nov 02 17:32:28.579517 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.579407 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr9k9\" (UniqueName: \"kubernetes.io/projected/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-kube-api-access-kr9k9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4\" (UID: \"d50aed5b-c7b1-4ac2-bb46-d5013752af0e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" Nov 02 17:32:28.579517 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.579435 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d32a6246-abfe-40fa-921d-560e9443aeb0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9\" (UID: \"d32a6246-abfe-40fa-921d-560e9443aeb0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9" Nov 02 17:32:28.579517 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.579474 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zrz5\" (UniqueName: \"kubernetes.io/projected/d32a6246-abfe-40fa-921d-560e9443aeb0-kube-api-access-9zrz5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9\" (UID: \"d32a6246-abfe-40fa-921d-560e9443aeb0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9" Nov 02 17:32:28.579730 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.579540 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4\" (UID: \"d50aed5b-c7b1-4ac2-bb46-d5013752af0e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" Nov 02 17:32:28.579940 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.579912 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4\" (UID: \"d50aed5b-c7b1-4ac2-bb46-d5013752af0e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" Nov 02 17:32:28.580048 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.579968 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4\" (UID: \"d50aed5b-c7b1-4ac2-bb46-d5013752af0e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" Nov 02 17:32:28.580411 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.580176 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d32a6246-abfe-40fa-921d-560e9443aeb0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9\" (UID: \"d32a6246-abfe-40fa-921d-560e9443aeb0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9" Nov 02 17:32:28.580411 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.580380 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d32a6246-abfe-40fa-921d-560e9443aeb0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9\" (UID: \"d32a6246-abfe-40fa-921d-560e9443aeb0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9" Nov 02 17:32:28.603354 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.603320 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zrz5\" (UniqueName: \"kubernetes.io/projected/d32a6246-abfe-40fa-921d-560e9443aeb0-kube-api-access-9zrz5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9\" (UID: \"d32a6246-abfe-40fa-921d-560e9443aeb0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9" Nov 02 17:32:28.603482 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.603464 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr9k9\" (UniqueName: \"kubernetes.io/projected/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-kube-api-access-kr9k9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4\" (UID: \"d50aed5b-c7b1-4ac2-bb46-d5013752af0e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" Nov 02 17:32:28.689796 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.689769 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9" Nov 02 17:32:28.698023 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.697997 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q" Nov 02 17:32:28.799197 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.799170 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd"] Nov 02 17:32:28.801249 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:32:28.801222 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e11a3a4_96e3_42da_905a_4ccb26fde15e.slice/crio-25be812c5069800d0cb72925b061886c7be3b7577ceb1ec43153483dee37320e WatchSource:0}: Error finding container 25be812c5069800d0cb72925b061886c7be3b7577ceb1ec43153483dee37320e: Status 404 returned error can't find the container with id 25be812c5069800d0cb72925b061886c7be3b7577ceb1ec43153483dee37320e Nov 02 17:32:28.898300 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:28.898266 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" Nov 02 17:32:29.000467 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.000440 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9"] Nov 02 17:32:29.003161 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:32:29.002817 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32a6246_abfe_40fa_921d_560e9443aeb0.slice/crio-b1803874ef54a6b80b7012cbe94afd7f6955edc3b56dcc36d9abb68654f53b9c WatchSource:0}: Error finding container b1803874ef54a6b80b7012cbe94afd7f6955edc3b56dcc36d9abb68654f53b9c: Status 404 returned error can't find the container with id b1803874ef54a6b80b7012cbe94afd7f6955edc3b56dcc36d9abb68654f53b9c Nov 02 17:32:29.007987 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.007961 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q"] Nov 02 17:32:29.010833 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:32:29.010810 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02198c6_d4e4_4bda_90e6_182f01fa068a.slice/crio-a3d61591f3d9ba4d7e6d026b3e5578ce1dbbe512c2932471ef4419fdada445af WatchSource:0}: Error finding container a3d61591f3d9ba4d7e6d026b3e5578ce1dbbe512c2932471ef4419fdada445af: Status 404 returned error can't find the container with id a3d61591f3d9ba4d7e6d026b3e5578ce1dbbe512c2932471ef4419fdada445af Nov 02 17:32:29.058587 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.058552 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt"] Nov 02 17:32:29.062465 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.062440 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" Nov 02 17:32:29.077914 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.077883 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-blm24\"" Nov 02 17:32:29.082717 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.082684 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt"] Nov 02 17:32:29.085919 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.085662 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6891e649-ad88-4bb8-80f3-8f671414df5a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt\" (UID: \"6891e649-ad88-4bb8-80f3-8f671414df5a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" Nov 02 17:32:29.085919 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.085722 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msgx9\" (UniqueName: \"kubernetes.io/projected/6891e649-ad88-4bb8-80f3-8f671414df5a-kube-api-access-msgx9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt\" (UID: \"6891e649-ad88-4bb8-80f3-8f671414df5a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" Nov 02 17:32:29.085919 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.085765 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6891e649-ad88-4bb8-80f3-8f671414df5a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt\" (UID: \"6891e649-ad88-4bb8-80f3-8f671414df5a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" Nov 02 17:32:29.187021 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.186980 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6891e649-ad88-4bb8-80f3-8f671414df5a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt\" (UID: \"6891e649-ad88-4bb8-80f3-8f671414df5a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" Nov 02 17:32:29.187191 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.187045 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msgx9\" (UniqueName: \"kubernetes.io/projected/6891e649-ad88-4bb8-80f3-8f671414df5a-kube-api-access-msgx9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt\" (UID: \"6891e649-ad88-4bb8-80f3-8f671414df5a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" Nov 02 17:32:29.187191 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.187087 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6891e649-ad88-4bb8-80f3-8f671414df5a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt\" (UID: \"6891e649-ad88-4bb8-80f3-8f671414df5a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" Nov 02 17:32:29.187510 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.187480 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6891e649-ad88-4bb8-80f3-8f671414df5a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt\" (UID: \"6891e649-ad88-4bb8-80f3-8f671414df5a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" Nov 02 17:32:29.187747 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.187728 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6891e649-ad88-4bb8-80f3-8f671414df5a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt\" (UID: \"6891e649-ad88-4bb8-80f3-8f671414df5a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" Nov 02 17:32:29.206681 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.206626 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4"] Nov 02 17:32:29.209660 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:32:29.209616 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50aed5b_c7b1_4ac2_bb46_d5013752af0e.slice/crio-36c16573cab6b8e0d4fefbc61324c8f5eb44c5845a474dc81f372dfc6b409e45 WatchSource:0}: Error finding container 36c16573cab6b8e0d4fefbc61324c8f5eb44c5845a474dc81f372dfc6b409e45: Status 404 returned error can't find the container with id 36c16573cab6b8e0d4fefbc61324c8f5eb44c5845a474dc81f372dfc6b409e45 Nov 02 17:32:29.213396 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.213333 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msgx9\" (UniqueName: \"kubernetes.io/projected/6891e649-ad88-4bb8-80f3-8f671414df5a-kube-api-access-msgx9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt\" (UID: \"6891e649-ad88-4bb8-80f3-8f671414df5a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" Nov 02 17:32:29.234085 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.234013 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd" event={"ID":"3e11a3a4-96e3-42da-905a-4ccb26fde15e","Type":"ContainerStarted","Data":"25be812c5069800d0cb72925b061886c7be3b7577ceb1ec43153483dee37320e"} Nov 02 17:32:29.236022 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.235976 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q" event={"ID":"c02198c6-d4e4-4bda-90e6-182f01fa068a","Type":"ContainerStarted","Data":"a3d61591f3d9ba4d7e6d026b3e5578ce1dbbe512c2932471ef4419fdada445af"} Nov 02 17:32:29.238692 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.238485 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" event={"ID":"d50aed5b-c7b1-4ac2-bb46-d5013752af0e","Type":"ContainerStarted","Data":"36c16573cab6b8e0d4fefbc61324c8f5eb44c5845a474dc81f372dfc6b409e45"} Nov 02 17:32:29.241132 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.241106 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9" event={"ID":"d32a6246-abfe-40fa-921d-560e9443aeb0","Type":"ContainerStarted","Data":"b1803874ef54a6b80b7012cbe94afd7f6955edc3b56dcc36d9abb68654f53b9c"} Nov 02 17:32:29.242588 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.242561 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x" event={"ID":"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb","Type":"ContainerStarted","Data":"6d781d50814c287f41988e94a64cdf7ce95998661ff62104d8a5164df6a00480"} Nov 02 17:32:29.388207 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.387617 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" Nov 02 17:32:29.728764 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:29.728677 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt"] Nov 02 17:32:29.737214 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:32:29.737178 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6891e649_ad88_4bb8_80f3_8f671414df5a.slice/crio-96c9ab798efcecdd5373ba8301e1e2a7fa745e9bfd9ee43ce1e525143a4d420e WatchSource:0}: Error finding container 96c9ab798efcecdd5373ba8301e1e2a7fa745e9bfd9ee43ce1e525143a4d420e: Status 404 returned error can't find the container with id 96c9ab798efcecdd5373ba8301e1e2a7fa745e9bfd9ee43ce1e525143a4d420e Nov 02 17:32:30.256898 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:30.256818 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" event={"ID":"6891e649-ad88-4bb8-80f3-8f671414df5a","Type":"ContainerStarted","Data":"96c9ab798efcecdd5373ba8301e1e2a7fa745e9bfd9ee43ce1e525143a4d420e"} Nov 02 17:32:31.723681 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.723041 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v"] Nov 02 17:32:31.728897 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.728425 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.751049 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.751016 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 17:32:31.751214 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.751117 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lkf8q\"" Nov 02 17:32:31.763615 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.763505 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v"] Nov 02 17:32:31.811541 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.810976 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b05251-7074-41bc-8994-c1051abee475-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.811541 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.811062 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jrsn\" (UniqueName: \"kubernetes.io/projected/62b05251-7074-41bc-8994-c1051abee475-kube-api-access-2jrsn\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.811541 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.811108 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.811541 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.811141 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b05251-7074-41bc-8994-c1051abee475-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.811541 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.811177 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.811541 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.811202 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.811541 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.811228 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b05251-7074-41bc-8994-c1051abee475-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.811541 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.811271 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.811541 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.811308 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.912446 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.912410 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.912624 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.912474 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b05251-7074-41bc-8994-c1051abee475-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.912624 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.912524 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jrsn\" (UniqueName: \"kubernetes.io/projected/62b05251-7074-41bc-8994-c1051abee475-kube-api-access-2jrsn\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.912624 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.912560 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.912624 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.912588 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b05251-7074-41bc-8994-c1051abee475-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.912828 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.912629 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.912828 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.912656 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.912828 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.912681 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b05251-7074-41bc-8994-c1051abee475-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.912828 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.912724 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.913502 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.913479 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.913739 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.913722 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.914351 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.914327 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.914351 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.914335 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b05251-7074-41bc-8994-c1051abee475-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.914674 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.914589 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.917733 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.917683 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.918578 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.918505 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b05251-7074-41bc-8994-c1051abee475-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.955562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.955187 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b05251-7074-41bc-8994-c1051abee475-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.955562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.955553 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jrsn\" (UniqueName: \"kubernetes.io/projected/62b05251-7074-41bc-8994-c1051abee475-kube-api-access-2jrsn\") pod \"maas-default-gateway-openshift-default-76f4876667-v9l5v\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:31.963657 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.963624 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn"] Nov 02 17:32:31.970540 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.970478 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:31.991995 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.989904 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr"] Nov 02 17:32:31.994668 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.994620 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-g68s5\"" Nov 02 17:32:31.998424 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.998403 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn"] Nov 02 17:32:31.998540 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:31.998520 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr" Nov 02 17:32:32.013753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.013378 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc64162d-415e-48ce-bfe3-8238c020fe58-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.013753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.013442 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc64162d-415e-48ce-bfe3-8238c020fe58-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.013753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.013474 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc64162d-415e-48ce-bfe3-8238c020fe58-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.013753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.013503 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc64162d-415e-48ce-bfe3-8238c020fe58-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.013753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.013537 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tql4p\" (UniqueName: \"kubernetes.io/projected/bc64162d-415e-48ce-bfe3-8238c020fe58-kube-api-access-tql4p\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.013753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.013562 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4dec5066-dab8-480c-946e-81c8772eacf0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr\" (UID: \"4dec5066-dab8-480c-946e-81c8772eacf0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr" Nov 02 17:32:32.013753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.013593 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc64162d-415e-48ce-bfe3-8238c020fe58-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.013753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.013623 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dec5066-dab8-480c-946e-81c8772eacf0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr\" (UID: \"4dec5066-dab8-480c-946e-81c8772eacf0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr" Nov 02 17:32:32.013753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.013689 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc64162d-415e-48ce-bfe3-8238c020fe58-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.013753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.013717 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w2qq\" (UniqueName: \"kubernetes.io/projected/4dec5066-dab8-480c-946e-81c8772eacf0-kube-api-access-5w2qq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr\" (UID: \"4dec5066-dab8-480c-946e-81c8772eacf0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr" Nov 02 17:32:32.013753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.013749 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc64162d-415e-48ce-bfe3-8238c020fe58-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.014315 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.013776 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc64162d-415e-48ce-bfe3-8238c020fe58-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.033903 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.033834 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr"] Nov 02 17:32:32.047188 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.046923 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:32.114361 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.114329 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc64162d-415e-48ce-bfe3-8238c020fe58-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.114519 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.114380 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc64162d-415e-48ce-bfe3-8238c020fe58-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.114519 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.114408 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc64162d-415e-48ce-bfe3-8238c020fe58-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.114519 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.114433 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc64162d-415e-48ce-bfe3-8238c020fe58-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.114519 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.114458 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tql4p\" (UniqueName: \"kubernetes.io/projected/bc64162d-415e-48ce-bfe3-8238c020fe58-kube-api-access-tql4p\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.114519 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.114482 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4dec5066-dab8-480c-946e-81c8772eacf0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr\" (UID: \"4dec5066-dab8-480c-946e-81c8772eacf0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr" Nov 02 17:32:32.114519 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.114507 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc64162d-415e-48ce-bfe3-8238c020fe58-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.114835 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.114531 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dec5066-dab8-480c-946e-81c8772eacf0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr\" (UID: \"4dec5066-dab8-480c-946e-81c8772eacf0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr" Nov 02 17:32:32.114835 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.114592 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc64162d-415e-48ce-bfe3-8238c020fe58-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.114835 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.114621 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w2qq\" (UniqueName: \"kubernetes.io/projected/4dec5066-dab8-480c-946e-81c8772eacf0-kube-api-access-5w2qq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr\" (UID: \"4dec5066-dab8-480c-946e-81c8772eacf0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr" Nov 02 17:32:32.114835 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.114657 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc64162d-415e-48ce-bfe3-8238c020fe58-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.114835 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.114686 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc64162d-415e-48ce-bfe3-8238c020fe58-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.115629 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.115520 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4dec5066-dab8-480c-946e-81c8772eacf0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr\" (UID: \"4dec5066-dab8-480c-946e-81c8772eacf0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr" Nov 02 17:32:32.115629 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.115598 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dec5066-dab8-480c-946e-81c8772eacf0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr\" (UID: \"4dec5066-dab8-480c-946e-81c8772eacf0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr" Nov 02 17:32:32.116304 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.116277 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc64162d-415e-48ce-bfe3-8238c020fe58-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.116527 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.116507 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc64162d-415e-48ce-bfe3-8238c020fe58-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.117997 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.117033 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc64162d-415e-48ce-bfe3-8238c020fe58-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.117997 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.117287 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc64162d-415e-48ce-bfe3-8238c020fe58-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.117997 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.117514 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc64162d-415e-48ce-bfe3-8238c020fe58-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.123708 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.123647 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc64162d-415e-48ce-bfe3-8238c020fe58-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.134673 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.134602 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc64162d-415e-48ce-bfe3-8238c020fe58-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.146542 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.146476 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc64162d-415e-48ce-bfe3-8238c020fe58-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.150363 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.150261 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tql4p\" (UniqueName: \"kubernetes.io/projected/bc64162d-415e-48ce-bfe3-8238c020fe58-kube-api-access-tql4p\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn\" (UID: \"bc64162d-415e-48ce-bfe3-8238c020fe58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.151221 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.151152 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w2qq\" (UniqueName: \"kubernetes.io/projected/4dec5066-dab8-480c-946e-81c8772eacf0-kube-api-access-5w2qq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr\" (UID: \"4dec5066-dab8-480c-946e-81c8772eacf0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr" Nov 02 17:32:32.298878 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.298443 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:32.321042 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.320987 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr" Nov 02 17:32:32.426992 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.426783 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v"] Nov 02 17:32:32.439580 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:32:32.439327 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b05251_7074_41bc_8994_c1051abee475.slice/crio-a176ffc3ede0c1545a4eb59f8a5abc48b18bf2cffce20473c05058117126a545 WatchSource:0}: Error finding container a176ffc3ede0c1545a4eb59f8a5abc48b18bf2cffce20473c05058117126a545: Status 404 returned error can't find the container with id a176ffc3ede0c1545a4eb59f8a5abc48b18bf2cffce20473c05058117126a545 Nov 02 17:32:32.678223 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.678158 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn"] Nov 02 17:32:32.684280 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:32:32.684242 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc64162d_415e_48ce_bfe3_8238c020fe58.slice/crio-bfea5c6ae140d0d1a99ad7137b428140b2b8a862bdfa8caf615805d895d5b817 WatchSource:0}: Error finding container bfea5c6ae140d0d1a99ad7137b428140b2b8a862bdfa8caf615805d895d5b817: Status 404 returned error can't find the container with id bfea5c6ae140d0d1a99ad7137b428140b2b8a862bdfa8caf615805d895d5b817 Nov 02 17:32:32.702770 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:32.702736 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr"] Nov 02 17:32:32.714009 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:32:32.713961 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dec5066_dab8_480c_946e_81c8772eacf0.slice/crio-c41e33a0208bb7a1e1ff6d961ed6f3833e4862786d8af96dbdb29807979745b6 WatchSource:0}: Error finding container c41e33a0208bb7a1e1ff6d961ed6f3833e4862786d8af96dbdb29807979745b6: Status 404 returned error can't find the container with id c41e33a0208bb7a1e1ff6d961ed6f3833e4862786d8af96dbdb29807979745b6 Nov 02 17:32:33.293483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:33.293449 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" event={"ID":"62b05251-7074-41bc-8994-c1051abee475","Type":"ContainerStarted","Data":"a176ffc3ede0c1545a4eb59f8a5abc48b18bf2cffce20473c05058117126a545"} Nov 02 17:32:33.299817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:33.299729 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" event={"ID":"bc64162d-415e-48ce-bfe3-8238c020fe58","Type":"ContainerStarted","Data":"bfea5c6ae140d0d1a99ad7137b428140b2b8a862bdfa8caf615805d895d5b817"} Nov 02 17:32:33.311118 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:33.311071 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr" event={"ID":"4dec5066-dab8-480c-946e-81c8772eacf0","Type":"ContainerStarted","Data":"c41e33a0208bb7a1e1ff6d961ed6f3833e4862786d8af96dbdb29807979745b6"} Nov 02 17:32:35.936166 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:35.936078 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd"] Nov 02 17:32:35.941676 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:35.941649 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd" Nov 02 17:32:35.974606 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:35.974358 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d633e91f-6217-432f-940e-36ba5fae132e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd\" (UID: \"d633e91f-6217-432f-940e-36ba5fae132e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd" Nov 02 17:32:35.974606 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:35.974423 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khnws\" (UniqueName: \"kubernetes.io/projected/d633e91f-6217-432f-940e-36ba5fae132e-kube-api-access-khnws\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd\" (UID: \"d633e91f-6217-432f-940e-36ba5fae132e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd" Nov 02 17:32:35.974606 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:35.974482 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d633e91f-6217-432f-940e-36ba5fae132e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd\" (UID: \"d633e91f-6217-432f-940e-36ba5fae132e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd" Nov 02 17:32:35.977067 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:35.977032 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd"] Nov 02 17:32:35.979230 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:35.979202 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq"] Nov 02 17:32:35.985545 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:35.985519 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" Nov 02 17:32:36.002567 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.002492 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq"] Nov 02 17:32:36.017000 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.015656 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls"] Nov 02 17:32:36.021127 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.021103 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" Nov 02 17:32:36.042593 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.042561 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls"] Nov 02 17:32:36.061110 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.059938 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns"] Nov 02 17:32:36.065471 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.065257 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns" Nov 02 17:32:36.075934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.075900 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d633e91f-6217-432f-940e-36ba5fae132e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd\" (UID: \"d633e91f-6217-432f-940e-36ba5fae132e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd" Nov 02 17:32:36.076089 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.075960 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/963e8558-7c81-45fa-96eb-b8e4e11d73f4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq\" (UID: \"963e8558-7c81-45fa-96eb-b8e4e11d73f4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" Nov 02 17:32:36.076089 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.076033 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/963e8558-7c81-45fa-96eb-b8e4e11d73f4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq\" (UID: \"963e8558-7c81-45fa-96eb-b8e4e11d73f4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" Nov 02 17:32:36.076089 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.076060 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls\" (UID: \"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" Nov 02 17:32:36.076258 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.076103 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t58f\" (UniqueName: \"kubernetes.io/projected/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-kube-api-access-6t58f\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls\" (UID: \"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" Nov 02 17:32:36.076258 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.076148 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d633e91f-6217-432f-940e-36ba5fae132e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd\" (UID: \"d633e91f-6217-432f-940e-36ba5fae132e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd" Nov 02 17:32:36.076258 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.076189 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khnws\" (UniqueName: \"kubernetes.io/projected/d633e91f-6217-432f-940e-36ba5fae132e-kube-api-access-khnws\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd\" (UID: \"d633e91f-6217-432f-940e-36ba5fae132e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd" Nov 02 17:32:36.076258 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.076220 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls\" (UID: \"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" Nov 02 17:32:36.076258 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.076255 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mlnf\" (UniqueName: \"kubernetes.io/projected/963e8558-7c81-45fa-96eb-b8e4e11d73f4-kube-api-access-7mlnf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq\" (UID: \"963e8558-7c81-45fa-96eb-b8e4e11d73f4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" Nov 02 17:32:36.076860 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.076707 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d633e91f-6217-432f-940e-36ba5fae132e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd\" (UID: \"d633e91f-6217-432f-940e-36ba5fae132e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd" Nov 02 17:32:36.077370 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.077344 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d633e91f-6217-432f-940e-36ba5fae132e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd\" (UID: \"d633e91f-6217-432f-940e-36ba5fae132e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd" Nov 02 17:32:36.085999 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.085974 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns"] Nov 02 17:32:36.101880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.101828 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khnws\" (UniqueName: \"kubernetes.io/projected/d633e91f-6217-432f-940e-36ba5fae132e-kube-api-access-khnws\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd\" (UID: \"d633e91f-6217-432f-940e-36ba5fae132e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd" Nov 02 17:32:36.179218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.176964 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t58f\" (UniqueName: \"kubernetes.io/projected/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-kube-api-access-6t58f\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls\" (UID: \"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" Nov 02 17:32:36.179218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.177043 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls\" (UID: \"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" Nov 02 17:32:36.179218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.177076 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mlnf\" (UniqueName: \"kubernetes.io/projected/963e8558-7c81-45fa-96eb-b8e4e11d73f4-kube-api-access-7mlnf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq\" (UID: \"963e8558-7c81-45fa-96eb-b8e4e11d73f4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" Nov 02 17:32:36.179218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.177138 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g44lr\" (UniqueName: \"kubernetes.io/projected/7f4595dc-331a-460e-94e8-d3395391c57b-kube-api-access-g44lr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns\" (UID: \"7f4595dc-331a-460e-94e8-d3395391c57b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns" Nov 02 17:32:36.179218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.177177 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/963e8558-7c81-45fa-96eb-b8e4e11d73f4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq\" (UID: \"963e8558-7c81-45fa-96eb-b8e4e11d73f4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" Nov 02 17:32:36.179218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.177203 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f4595dc-331a-460e-94e8-d3395391c57b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns\" (UID: \"7f4595dc-331a-460e-94e8-d3395391c57b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns" Nov 02 17:32:36.179218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.177255 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f4595dc-331a-460e-94e8-d3395391c57b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns\" (UID: \"7f4595dc-331a-460e-94e8-d3395391c57b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns" Nov 02 17:32:36.179218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.177297 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/963e8558-7c81-45fa-96eb-b8e4e11d73f4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq\" (UID: \"963e8558-7c81-45fa-96eb-b8e4e11d73f4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" Nov 02 17:32:36.179218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.177322 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls\" (UID: \"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" Nov 02 17:32:36.179218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.177734 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls\" (UID: \"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" Nov 02 17:32:36.179218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.178342 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls\" (UID: \"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" Nov 02 17:32:36.179218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.178900 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/963e8558-7c81-45fa-96eb-b8e4e11d73f4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq\" (UID: \"963e8558-7c81-45fa-96eb-b8e4e11d73f4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" Nov 02 17:32:36.179218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.179169 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/963e8558-7c81-45fa-96eb-b8e4e11d73f4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq\" (UID: \"963e8558-7c81-45fa-96eb-b8e4e11d73f4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" Nov 02 17:32:36.210982 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.210754 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t58f\" (UniqueName: \"kubernetes.io/projected/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-kube-api-access-6t58f\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls\" (UID: \"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" Nov 02 17:32:36.214596 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.214565 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mlnf\" (UniqueName: \"kubernetes.io/projected/963e8558-7c81-45fa-96eb-b8e4e11d73f4-kube-api-access-7mlnf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq\" (UID: \"963e8558-7c81-45fa-96eb-b8e4e11d73f4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" Nov 02 17:32:36.268494 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.268086 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd" Nov 02 17:32:36.280040 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.278757 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g44lr\" (UniqueName: \"kubernetes.io/projected/7f4595dc-331a-460e-94e8-d3395391c57b-kube-api-access-g44lr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns\" (UID: \"7f4595dc-331a-460e-94e8-d3395391c57b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns" Nov 02 17:32:36.280040 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.278809 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f4595dc-331a-460e-94e8-d3395391c57b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns\" (UID: \"7f4595dc-331a-460e-94e8-d3395391c57b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns" Nov 02 17:32:36.280040 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.278874 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f4595dc-331a-460e-94e8-d3395391c57b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns\" (UID: \"7f4595dc-331a-460e-94e8-d3395391c57b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns" Nov 02 17:32:36.280040 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.279313 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f4595dc-331a-460e-94e8-d3395391c57b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns\" (UID: \"7f4595dc-331a-460e-94e8-d3395391c57b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns" Nov 02 17:32:36.280040 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.279902 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f4595dc-331a-460e-94e8-d3395391c57b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns\" (UID: \"7f4595dc-331a-460e-94e8-d3395391c57b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns" Nov 02 17:32:36.316178 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.316137 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g44lr\" (UniqueName: \"kubernetes.io/projected/7f4595dc-331a-460e-94e8-d3395391c57b-kube-api-access-g44lr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns\" (UID: \"7f4595dc-331a-460e-94e8-d3395391c57b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns" Nov 02 17:32:36.316355 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.316332 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" Nov 02 17:32:36.347119 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.346860 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" Nov 02 17:32:36.402394 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.402003 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns" Nov 02 17:32:36.660241 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.660204 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd"] Nov 02 17:32:36.666895 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:32:36.666859 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd633e91f_6217_432f_940e_36ba5fae132e.slice/crio-73d147777ded21287e2d07bddccfedc7cdce05b4a450a5405e54044ccce301d8 WatchSource:0}: Error finding container 73d147777ded21287e2d07bddccfedc7cdce05b4a450a5405e54044ccce301d8: Status 404 returned error can't find the container with id 73d147777ded21287e2d07bddccfedc7cdce05b4a450a5405e54044ccce301d8 Nov 02 17:32:36.682977 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.682397 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq"] Nov 02 17:32:36.710438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.710417 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls"] Nov 02 17:32:36.761035 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:36.760900 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns"] Nov 02 17:32:36.766716 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:32:36.766654 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4595dc_331a_460e_94e8_d3395391c57b.slice/crio-6f68ece541630a631f2abc448b1eb5785ba9d37a71ba0e63b746228a7c63d79d WatchSource:0}: Error finding container 6f68ece541630a631f2abc448b1eb5785ba9d37a71ba0e63b746228a7c63d79d: Status 404 returned error can't find the container with id 6f68ece541630a631f2abc448b1eb5785ba9d37a71ba0e63b746228a7c63d79d Nov 02 17:32:37.394722 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:37.394109 4860 generic.go:356] "Generic (PLEG): container finished" podID="0caba324-7d75-479c-a1ca-1af2ec3819cb" containerID="33ef38cf0610fe3b87018c2abfe80f0626a235f91b94c181b083337175f0020d" exitCode=0 Nov 02 17:32:37.394722 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:37.394237 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-86mkk-5hzgx" event={"ID":"0caba324-7d75-479c-a1ca-1af2ec3819cb","Type":"ContainerDied","Data":"33ef38cf0610fe3b87018c2abfe80f0626a235f91b94c181b083337175f0020d"} Nov 02 17:32:37.398052 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:37.397927 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd" event={"ID":"d633e91f-6217-432f-940e-36ba5fae132e","Type":"ContainerStarted","Data":"73d147777ded21287e2d07bddccfedc7cdce05b4a450a5405e54044ccce301d8"} Nov 02 17:32:37.421180 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:37.421130 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns" event={"ID":"7f4595dc-331a-460e-94e8-d3395391c57b","Type":"ContainerStarted","Data":"6f68ece541630a631f2abc448b1eb5785ba9d37a71ba0e63b746228a7c63d79d"} Nov 02 17:32:37.426865 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:37.426810 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" event={"ID":"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc","Type":"ContainerStarted","Data":"4283d2f179f24be136c7305c6cfba0c07a954a081b7df4a8db2198bd912088bf"} Nov 02 17:32:37.445433 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:37.445395 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" event={"ID":"963e8558-7c81-45fa-96eb-b8e4e11d73f4","Type":"ContainerStarted","Data":"e40d4b2012513784eadb1d38c458674d8aa2ec7636d01e5083557a07589abb65"} Nov 02 17:32:38.733678 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:38.733359 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-86mkk-5hzgx" Nov 02 17:32:38.811999 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:38.810991 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0caba324-7d75-479c-a1ca-1af2ec3819cb-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0caba324-7d75-479c-a1ca-1af2ec3819cb-archives-path\") pod \"0caba324-7d75-479c-a1ca-1af2ec3819cb\" (UID: \"0caba324-7d75-479c-a1ca-1af2ec3819cb\") " Nov 02 17:32:38.811999 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:38.811062 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0caba324-7d75-479c-a1ca-1af2ec3819cb-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0caba324-7d75-479c-a1ca-1af2ec3819cb-service-ca-bundle\") pod \"0caba324-7d75-479c-a1ca-1af2ec3819cb\" (UID: \"0caba324-7d75-479c-a1ca-1af2ec3819cb\") " Nov 02 17:32:38.811999 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:38.811128 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0caba324-7d75-479c-a1ca-1af2ec3819cb-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0caba324-7d75-479c-a1ca-1af2ec3819cb-trusted-ca-bundle\") pod \"0caba324-7d75-479c-a1ca-1af2ec3819cb\" (UID: \"0caba324-7d75-479c-a1ca-1af2ec3819cb\") " Nov 02 17:32:38.811999 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:38.811182 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0caba324-7d75-479c-a1ca-1af2ec3819cb-kube-api-access-5vfkp\" (UniqueName: \"kubernetes.io/projected/0caba324-7d75-479c-a1ca-1af2ec3819cb-kube-api-access-5vfkp\") pod \"0caba324-7d75-479c-a1ca-1af2ec3819cb\" (UID: \"0caba324-7d75-479c-a1ca-1af2ec3819cb\") " Nov 02 17:32:38.812370 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:38.812332 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0caba324-7d75-479c-a1ca-1af2ec3819cb-archives-path" pod "0caba324-7d75-479c-a1ca-1af2ec3819cb" (UID: "0caba324-7d75-479c-a1ca-1af2ec3819cb"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:32:38.812809 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:38.812738 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0caba324-7d75-479c-a1ca-1af2ec3819cb-trusted-ca-bundle" pod "0caba324-7d75-479c-a1ca-1af2ec3819cb" (UID: "0caba324-7d75-479c-a1ca-1af2ec3819cb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:32:38.812809 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:38.812746 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0caba324-7d75-479c-a1ca-1af2ec3819cb-service-ca-bundle" pod "0caba324-7d75-479c-a1ca-1af2ec3819cb" (UID: "0caba324-7d75-479c-a1ca-1af2ec3819cb"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:32:38.846346 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:38.846287 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0caba324-7d75-479c-a1ca-1af2ec3819cb-kube-api-access-5vfkp" pod "0caba324-7d75-479c-a1ca-1af2ec3819cb" (UID: "0caba324-7d75-479c-a1ca-1af2ec3819cb"). InnerVolumeSpecName "kube-api-access-5vfkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:32:38.912554 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:38.912462 4860 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0caba324-7d75-479c-a1ca-1af2ec3819cb-trusted-ca-bundle\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:32:38.912554 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:38.912497 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vfkp\" (UniqueName: \"kubernetes.io/projected/0caba324-7d75-479c-a1ca-1af2ec3819cb-kube-api-access-5vfkp\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:32:38.912554 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:38.912512 4860 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0caba324-7d75-479c-a1ca-1af2ec3819cb-archives-path\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:32:38.912554 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:38.912527 4860 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0caba324-7d75-479c-a1ca-1af2ec3819cb-service-ca-bundle\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:32:39.494876 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:39.491029 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-86mkk-5hzgx" event={"ID":"0caba324-7d75-479c-a1ca-1af2ec3819cb","Type":"ContainerDied","Data":"ef0d2ef89d4bf6e9880f885caf8efd7b71ac06bab4d2f288cf594fd4b8141684"} Nov 02 17:32:39.494876 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:39.491105 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef0d2ef89d4bf6e9880f885caf8efd7b71ac06bab4d2f288cf594fd4b8141684" Nov 02 17:32:39.494876 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:39.494569 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-86mkk-5hzgx" Nov 02 17:32:44.049160 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:44.049094 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:32:44.049649 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:44.049389 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:32:44.049649 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:44.049464 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:32:44.467760 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:44.467694 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:32:44.467946 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:44.467798 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:32:44.467946 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:44.467844 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:32:44.585153 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:44.584111 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" event={"ID":"62b05251-7074-41bc-8994-c1051abee475","Type":"ContainerStarted","Data":"afeda81a9e268826c31617880458f51bce447ac95759eea3a4e619b0cd6115a9"} Nov 02 17:32:45.048197 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:45.048163 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:45.054361 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:45.054102 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:45.103347 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:45.102998 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" podStartSLOduration=2.5082662300000003 podStartE2EDuration="14.102977282s" totalImagesPullingTime="11.594711052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:44.646637805 +0000 UTC m=+183.267997587" watchObservedRunningTime="2026-11-02 17:32:45.102977282 +0000 UTC m=+183.724337059" Nov 02 17:32:45.606782 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:45.606697 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" event={"ID":"bc64162d-415e-48ce-bfe3-8238c020fe58","Type":"ContainerStarted","Data":"db64af8e44be76e78379545e51c82e443267613d2b0b7058fc50668c110389b8"} Nov 02 17:32:45.607524 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:45.607430 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:45.608670 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:45.608647 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:32:45.654933 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:45.654711 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" podStartSLOduration=2.881600549 podStartE2EDuration="14.654693983s" totalImagesPullingTime="11.773093434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:45.653151437 +0000 UTC m=+184.274511215" watchObservedRunningTime="2026-11-02 17:32:45.654693983 +0000 UTC m=+184.276053758" Nov 02 17:32:46.299590 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:46.299557 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:46.306469 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:46.306435 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:46.621598 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:46.621566 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:46.622988 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:46.622928 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn" Nov 02 17:32:48.668119 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:48.667341 4860 generic.go:356] "Generic (PLEG): container finished" podID="565e578b-d2dc-4fb3-987f-70d19da8148c" containerID="7af9c55dace0b955418b3cc475a67865cdd777e5ed7c9b82df391fc0da5cbce6" exitCode=0 Nov 02 17:32:48.668119 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:48.667427 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7" event={"ID":"565e578b-d2dc-4fb3-987f-70d19da8148c","Type":"ContainerDied","Data":"7af9c55dace0b955418b3cc475a67865cdd777e5ed7c9b82df391fc0da5cbce6"} Nov 02 17:32:48.683249 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:48.683211 4860 generic.go:356] "Generic (PLEG): container finished" podID="4dec5066-dab8-480c-946e-81c8772eacf0" containerID="da8fc2bcef8195232d1f0102661f6e362cb06c24dcb582b6e858d20a8feeb776" exitCode=0 Nov 02 17:32:48.684719 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:48.684693 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr" event={"ID":"4dec5066-dab8-480c-946e-81c8772eacf0","Type":"ContainerDied","Data":"da8fc2bcef8195232d1f0102661f6e362cb06c24dcb582b6e858d20a8feeb776"} Nov 02 17:32:51.716060 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:51.716020 4860 generic.go:356] "Generic (PLEG): container finished" podID="d32a6246-abfe-40fa-921d-560e9443aeb0" containerID="766eb9f6e8216f81b17b8d0cf3e07e9ffaaa1b9dba3fffcea2f2a6647c87ae14" exitCode=0 Nov 02 17:32:51.716600 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:51.716099 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9" event={"ID":"d32a6246-abfe-40fa-921d-560e9443aeb0","Type":"ContainerDied","Data":"766eb9f6e8216f81b17b8d0cf3e07e9ffaaa1b9dba3fffcea2f2a6647c87ae14"} Nov 02 17:32:51.717827 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:51.717797 4860 generic.go:356] "Generic (PLEG): container finished" podID="062ebaf9-1d88-41af-9d5e-5b233f2a1eeb" containerID="b594fe543d5b77ad5330d7901a7eed430e60be3a1ac6869c4a9df1288f89c03d" exitCode=0 Nov 02 17:32:51.717954 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:51.717878 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x" event={"ID":"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb","Type":"ContainerDied","Data":"b594fe543d5b77ad5330d7901a7eed430e60be3a1ac6869c4a9df1288f89c03d"} Nov 02 17:32:51.719389 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:51.719369 4860 generic.go:356] "Generic (PLEG): container finished" podID="963e8558-7c81-45fa-96eb-b8e4e11d73f4" containerID="eba22a6035bd36c7922767ac7d7a351db17ae94987b023e079ad4a92ae9c4d7a" exitCode=0 Nov 02 17:32:51.719478 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:51.719447 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" event={"ID":"963e8558-7c81-45fa-96eb-b8e4e11d73f4","Type":"ContainerDied","Data":"eba22a6035bd36c7922767ac7d7a351db17ae94987b023e079ad4a92ae9c4d7a"} Nov 02 17:32:51.720948 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:51.720925 4860 generic.go:356] "Generic (PLEG): container finished" podID="c02198c6-d4e4-4bda-90e6-182f01fa068a" containerID="c129a5c4f0337d5213e82e7d2f7da50d212cbe4b62b57f150232d306bf77ce89" exitCode=0 Nov 02 17:32:51.721063 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:51.720962 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q" event={"ID":"c02198c6-d4e4-4bda-90e6-182f01fa068a","Type":"ContainerDied","Data":"c129a5c4f0337d5213e82e7d2f7da50d212cbe4b62b57f150232d306bf77ce89"} Nov 02 17:32:51.771224 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:32:51.771050 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:32:52.729784 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:52.729742 4860 generic.go:356] "Generic (PLEG): container finished" podID="d50aed5b-c7b1-4ac2-bb46-d5013752af0e" containerID="f1116d032beaf4d7bed17a80d4c4edac8e9b63af0ba34163f690a1caf728a656" exitCode=0 Nov 02 17:32:52.730588 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:52.730523 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" event={"ID":"d50aed5b-c7b1-4ac2-bb46-d5013752af0e","Type":"ContainerDied","Data":"f1116d032beaf4d7bed17a80d4c4edac8e9b63af0ba34163f690a1caf728a656"} Nov 02 17:32:52.733704 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:52.733666 4860 generic.go:356] "Generic (PLEG): container finished" podID="fe8a0285-755a-4758-b3c9-8b93e0a9e8cc" containerID="f3e2b4a9e30b4cd24a3db6f3392f03440818431d2f4b58638b33d14467924a72" exitCode=0 Nov 02 17:32:52.733815 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:52.733799 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" event={"ID":"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc","Type":"ContainerDied","Data":"f3e2b4a9e30b4cd24a3db6f3392f03440818431d2f4b58638b33d14467924a72"} Nov 02 17:32:52.740073 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:52.738674 4860 generic.go:356] "Generic (PLEG): container finished" podID="6891e649-ad88-4bb8-80f3-8f671414df5a" containerID="07e820a588b18517291e2dcaa9138a15f0c771c03175f40e589f7a6c899ee90e" exitCode=0 Nov 02 17:32:52.740073 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:52.738749 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" event={"ID":"6891e649-ad88-4bb8-80f3-8f671414df5a","Type":"ContainerDied","Data":"07e820a588b18517291e2dcaa9138a15f0c771c03175f40e589f7a6c899ee90e"} Nov 02 17:32:53.745905 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:53.745798 4860 generic.go:356] "Generic (PLEG): container finished" podID="d32a6246-abfe-40fa-921d-560e9443aeb0" containerID="68a7e965b5d076a8a184640e5af82cf35ebf7e72476fe3238d7495e6c2d377d0" exitCode=0 Nov 02 17:32:53.745905 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:53.745886 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9" event={"ID":"d32a6246-abfe-40fa-921d-560e9443aeb0","Type":"ContainerDied","Data":"68a7e965b5d076a8a184640e5af82cf35ebf7e72476fe3238d7495e6c2d377d0"} Nov 02 17:32:53.748316 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:53.748264 4860 generic.go:356] "Generic (PLEG): container finished" podID="7f4595dc-331a-460e-94e8-d3395391c57b" containerID="2e15bd5d147041171ec1e4ed9b1e0f4fbb05a59659b0e156a37cad187e70beda" exitCode=0 Nov 02 17:32:53.748398 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:53.748348 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns" event={"ID":"7f4595dc-331a-460e-94e8-d3395391c57b","Type":"ContainerDied","Data":"2e15bd5d147041171ec1e4ed9b1e0f4fbb05a59659b0e156a37cad187e70beda"} Nov 02 17:32:53.750407 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:53.750364 4860 generic.go:356] "Generic (PLEG): container finished" podID="963e8558-7c81-45fa-96eb-b8e4e11d73f4" containerID="a8cd84f275d550dbb3f9599a9d275efee0c8b75b14ffe7f4eaa1d35d85ebfa58" exitCode=0 Nov 02 17:32:53.750477 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:53.750453 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" event={"ID":"963e8558-7c81-45fa-96eb-b8e4e11d73f4","Type":"ContainerDied","Data":"a8cd84f275d550dbb3f9599a9d275efee0c8b75b14ffe7f4eaa1d35d85ebfa58"} Nov 02 17:32:53.753384 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:53.753273 4860 generic.go:356] "Generic (PLEG): container finished" podID="6891e649-ad88-4bb8-80f3-8f671414df5a" containerID="ed454322d07f49708df9ddcbf25192b8450ef83fc0fa5711f9cb5d5f74ea782f" exitCode=0 Nov 02 17:32:53.753384 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:53.753337 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" event={"ID":"6891e649-ad88-4bb8-80f3-8f671414df5a","Type":"ContainerDied","Data":"ed454322d07f49708df9ddcbf25192b8450ef83fc0fa5711f9cb5d5f74ea782f"} Nov 02 17:32:53.755920 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:53.755894 4860 generic.go:356] "Generic (PLEG): container finished" podID="565e578b-d2dc-4fb3-987f-70d19da8148c" containerID="56ca7d31496fc01de6fd4a986c3fef42794f0a4c222cd0f9218ba3cd451a9926" exitCode=0 Nov 02 17:32:53.756010 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:53.755978 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7" event={"ID":"565e578b-d2dc-4fb3-987f-70d19da8148c","Type":"ContainerDied","Data":"56ca7d31496fc01de6fd4a986c3fef42794f0a4c222cd0f9218ba3cd451a9926"} Nov 02 17:32:53.757916 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:53.757766 4860 generic.go:356] "Generic (PLEG): container finished" podID="3e11a3a4-96e3-42da-905a-4ccb26fde15e" containerID="44fed0174e2f57394059751540eaa37d8e7e930a9dc3cbebb54b8e5ff7ed6c92" exitCode=0 Nov 02 17:32:53.758033 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:53.758016 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd" event={"ID":"3e11a3a4-96e3-42da-905a-4ccb26fde15e","Type":"ContainerDied","Data":"44fed0174e2f57394059751540eaa37d8e7e930a9dc3cbebb54b8e5ff7ed6c92"} Nov 02 17:32:53.760669 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:53.760498 4860 generic.go:356] "Generic (PLEG): container finished" podID="c02198c6-d4e4-4bda-90e6-182f01fa068a" containerID="72df87cd4c9b82e57ffd0f447270d5ef4752d3c78343c38a36194504b1673731" exitCode=0 Nov 02 17:32:53.760669 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:53.760621 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q" event={"ID":"c02198c6-d4e4-4bda-90e6-182f01fa068a","Type":"ContainerDied","Data":"72df87cd4c9b82e57ffd0f447270d5ef4752d3c78343c38a36194504b1673731"} Nov 02 17:32:53.762826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:53.762743 4860 generic.go:356] "Generic (PLEG): container finished" podID="4dec5066-dab8-480c-946e-81c8772eacf0" containerID="820cb741f9ae784f13fd1b3094b4180b5e6a4a185cc3f827ece2fbbfb3f809c0" exitCode=0 Nov 02 17:32:53.762920 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:53.762832 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr" event={"ID":"4dec5066-dab8-480c-946e-81c8772eacf0","Type":"ContainerDied","Data":"820cb741f9ae784f13fd1b3094b4180b5e6a4a185cc3f827ece2fbbfb3f809c0"} Nov 02 17:32:53.764702 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:53.764541 4860 generic.go:356] "Generic (PLEG): container finished" podID="d633e91f-6217-432f-940e-36ba5fae132e" containerID="5bfed9fe0689f027fc9159d505a64466e8bdaab8bad8c4afd10f292eb3ea4423" exitCode=0 Nov 02 17:32:53.764702 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:53.764578 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd" event={"ID":"d633e91f-6217-432f-940e-36ba5fae132e","Type":"ContainerDied","Data":"5bfed9fe0689f027fc9159d505a64466e8bdaab8bad8c4afd10f292eb3ea4423"} Nov 02 17:32:54.779896 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:54.779792 4860 generic.go:356] "Generic (PLEG): container finished" podID="3e11a3a4-96e3-42da-905a-4ccb26fde15e" containerID="c496a3048a2e690051a04801c9894fb2d21c37a6fe815db6cd2bee215e4b3f8d" exitCode=0 Nov 02 17:32:54.780342 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:54.779930 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd" event={"ID":"3e11a3a4-96e3-42da-905a-4ccb26fde15e","Type":"ContainerDied","Data":"c496a3048a2e690051a04801c9894fb2d21c37a6fe815db6cd2bee215e4b3f8d"} Nov 02 17:32:54.786744 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:54.786715 4860 generic.go:356] "Generic (PLEG): container finished" podID="d633e91f-6217-432f-940e-36ba5fae132e" containerID="8fff68675c275c69dc0dd7e5acd15fa4409ddb921d5c941836ad8ff6afa86483" exitCode=0 Nov 02 17:32:54.786921 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:54.786900 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd" event={"ID":"d633e91f-6217-432f-940e-36ba5fae132e","Type":"ContainerDied","Data":"8fff68675c275c69dc0dd7e5acd15fa4409ddb921d5c941836ad8ff6afa86483"} Nov 02 17:32:54.793511 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:54.793445 4860 generic.go:356] "Generic (PLEG): container finished" podID="7f4595dc-331a-460e-94e8-d3395391c57b" containerID="cf99742e419ffa3104ae61ff35ec950bc3ca07a1fee414fd1e21b9344142a9d8" exitCode=0 Nov 02 17:32:54.793608 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:54.793530 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns" event={"ID":"7f4595dc-331a-460e-94e8-d3395391c57b","Type":"ContainerDied","Data":"cf99742e419ffa3104ae61ff35ec950bc3ca07a1fee414fd1e21b9344142a9d8"} Nov 02 17:32:56.826887 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:56.821521 4860 generic.go:356] "Generic (PLEG): container finished" podID="062ebaf9-1d88-41af-9d5e-5b233f2a1eeb" containerID="6bd557fde56a98d1f967e1089b757d33381a4327af822faa5aa35dc03a25ab95" exitCode=0 Nov 02 17:32:56.826887 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:56.821638 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x" event={"ID":"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb","Type":"ContainerDied","Data":"6bd557fde56a98d1f967e1089b757d33381a4327af822faa5aa35dc03a25ab95"} Nov 02 17:32:56.833498 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:56.833465 4860 generic.go:356] "Generic (PLEG): container finished" podID="fe8a0285-755a-4758-b3c9-8b93e0a9e8cc" containerID="7ba2362f51777363ce4b1aa078e0b3a961e386161570e64c7450c1806f91a78c" exitCode=0 Nov 02 17:32:56.833808 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:32:56.833776 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" event={"ID":"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc","Type":"ContainerDied","Data":"7ba2362f51777363ce4b1aa078e0b3a961e386161570e64c7450c1806f91a78c"} Nov 02 17:33:10.040153 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:10.038952 4860 generic.go:356] "Generic (PLEG): container finished" podID="d50aed5b-c7b1-4ac2-bb46-d5013752af0e" containerID="d0783efda5c67e1cf77147bcbd87f48ebdc75472219c5b490bcc7e8a6a02cb93" exitCode=0 Nov 02 17:33:10.040153 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:10.039007 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" event={"ID":"d50aed5b-c7b1-4ac2-bb46-d5013752af0e","Type":"ContainerDied","Data":"d0783efda5c67e1cf77147bcbd87f48ebdc75472219c5b490bcc7e8a6a02cb93"} Nov 02 17:33:17.154402 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:17.154362 4860 generic.go:356] "Generic (PLEG): container finished" podID="d32a6246-abfe-40fa-921d-560e9443aeb0" containerID="c18e60d8c1df7d64b3a06a1c3c2d35143c94fb26756c2740b19b80c496f4a791" exitCode=0 Nov 02 17:33:17.154880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:17.154482 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9" event={"ID":"d32a6246-abfe-40fa-921d-560e9443aeb0","Type":"ContainerDied","Data":"c18e60d8c1df7d64b3a06a1c3c2d35143c94fb26756c2740b19b80c496f4a791"} Nov 02 17:33:17.162539 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:17.161981 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" event={"ID":"963e8558-7c81-45fa-96eb-b8e4e11d73f4","Type":"ContainerStarted","Data":"614e926ce49e380327ff014428ea848639f39b118f77df5a6e8113038f982fd8"} Nov 02 17:33:17.172190 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:17.171567 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" event={"ID":"6891e649-ad88-4bb8-80f3-8f671414df5a","Type":"ContainerStarted","Data":"567b692c55d10a61ce16e4042d83acd776a8c084e955e76a062e8c62a732a343"} Nov 02 17:33:17.181117 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:17.181091 4860 generic.go:356] "Generic (PLEG): container finished" podID="565e578b-d2dc-4fb3-987f-70d19da8148c" containerID="4c70b846bfc9b510dfd5071260dc2666a51e69408fe26f1ed4d3fe658e54d845" exitCode=0 Nov 02 17:33:17.181498 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:17.181246 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7" event={"ID":"565e578b-d2dc-4fb3-987f-70d19da8148c","Type":"ContainerDied","Data":"4c70b846bfc9b510dfd5071260dc2666a51e69408fe26f1ed4d3fe658e54d845"} Nov 02 17:33:17.183748 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:17.183728 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 17:33:17.184469 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:17.184411 4860 generic.go:356] "Generic (PLEG): container finished" podID="4dec5066-dab8-480c-946e-81c8772eacf0" containerID="6da35790369d66f998863790dd5f45072b602c58591da1c5d31fb07e722428f7" exitCode=1 Nov 02 17:33:17.184538 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:17.184471 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr" event={"ID":"4dec5066-dab8-480c-946e-81c8772eacf0","Type":"ContainerDied","Data":"6da35790369d66f998863790dd5f45072b602c58591da1c5d31fb07e722428f7"} Nov 02 17:33:17.291441 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:17.291386 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" podStartSLOduration=2.081207912 podStartE2EDuration="48.291370724s" totalImagesPullingTime="46.210162812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:17.245027963 +0000 UTC m=+215.866387747" watchObservedRunningTime="2026-11-02 17:33:17.291370724 +0000 UTC m=+215.912730500" Nov 02 17:33:17.383900 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:17.383774 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" podStartSLOduration=3.769017689 podStartE2EDuration="42.383756696s" totalImagesPullingTime="38.614739007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:17.335040588 +0000 UTC m=+215.956400363" watchObservedRunningTime="2026-11-02 17:33:17.383756696 +0000 UTC m=+216.005116473" Nov 02 17:33:18.195547 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.195510 4860 generic.go:356] "Generic (PLEG): container finished" podID="963e8558-7c81-45fa-96eb-b8e4e11d73f4" containerID="614e926ce49e380327ff014428ea848639f39b118f77df5a6e8113038f982fd8" exitCode=0 Nov 02 17:33:18.196116 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.195634 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" event={"ID":"963e8558-7c81-45fa-96eb-b8e4e11d73f4","Type":"ContainerDied","Data":"614e926ce49e380327ff014428ea848639f39b118f77df5a6e8113038f982fd8"} Nov 02 17:33:18.199078 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.198953 4860 generic.go:356] "Generic (PLEG): container finished" podID="6891e649-ad88-4bb8-80f3-8f671414df5a" containerID="567b692c55d10a61ce16e4042d83acd776a8c084e955e76a062e8c62a732a343" exitCode=0 Nov 02 17:33:18.199078 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.199034 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" event={"ID":"6891e649-ad88-4bb8-80f3-8f671414df5a","Type":"ContainerDied","Data":"567b692c55d10a61ce16e4042d83acd776a8c084e955e76a062e8c62a732a343"} Nov 02 17:33:18.205555 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.204684 4860 generic.go:356] "Generic (PLEG): container finished" podID="c02198c6-d4e4-4bda-90e6-182f01fa068a" containerID="6748c1cd3887219fd58fdda41ff33c8cf84a8be02f3a7477b6efb30c4a0f9707" exitCode=0 Nov 02 17:33:18.205555 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.205138 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q" event={"ID":"c02198c6-d4e4-4bda-90e6-182f01fa068a","Type":"ContainerDied","Data":"6748c1cd3887219fd58fdda41ff33c8cf84a8be02f3a7477b6efb30c4a0f9707"} Nov 02 17:33:18.423764 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.423737 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 17:33:18.437495 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.437215 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr" Nov 02 17:33:18.485749 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.485712 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dec5066-dab8-480c-946e-81c8772eacf0-util\" (UniqueName: \"kubernetes.io/empty-dir/4dec5066-dab8-480c-946e-81c8772eacf0-util\") pod \"4dec5066-dab8-480c-946e-81c8772eacf0\" (UID: \"4dec5066-dab8-480c-946e-81c8772eacf0\") " Nov 02 17:33:18.485929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.485783 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dec5066-dab8-480c-946e-81c8772eacf0-kube-api-access-5w2qq\" (UniqueName: \"kubernetes.io/projected/4dec5066-dab8-480c-946e-81c8772eacf0-kube-api-access-5w2qq\") pod \"4dec5066-dab8-480c-946e-81c8772eacf0\" (UID: \"4dec5066-dab8-480c-946e-81c8772eacf0\") " Nov 02 17:33:18.485929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.485924 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dec5066-dab8-480c-946e-81c8772eacf0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dec5066-dab8-480c-946e-81c8772eacf0-bundle\") pod \"4dec5066-dab8-480c-946e-81c8772eacf0\" (UID: \"4dec5066-dab8-480c-946e-81c8772eacf0\") " Nov 02 17:33:18.495817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.489397 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dec5066-dab8-480c-946e-81c8772eacf0-kube-api-access-5w2qq" pod "4dec5066-dab8-480c-946e-81c8772eacf0" (UID: "4dec5066-dab8-480c-946e-81c8772eacf0"). InnerVolumeSpecName "kube-api-access-5w2qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:33:18.495817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.489944 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dec5066-dab8-480c-946e-81c8772eacf0-bundle" pod "4dec5066-dab8-480c-946e-81c8772eacf0" (UID: "4dec5066-dab8-480c-946e-81c8772eacf0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:18.495817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.495438 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dec5066-dab8-480c-946e-81c8772eacf0-util" pod "4dec5066-dab8-480c-946e-81c8772eacf0" (UID: "4dec5066-dab8-480c-946e-81c8772eacf0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:18.553007 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.552979 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9" Nov 02 17:33:18.568232 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.568201 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7" Nov 02 17:33:18.587637 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.587608 4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dec5066-dab8-480c-946e-81c8772eacf0-bundle\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:18.587637 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.587639 4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4dec5066-dab8-480c-946e-81c8772eacf0-util\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:18.587806 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.587648 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w2qq\" (UniqueName: \"kubernetes.io/projected/4dec5066-dab8-480c-946e-81c8772eacf0-kube-api-access-5w2qq\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:18.691243 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.688312 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565e578b-d2dc-4fb3-987f-70d19da8148c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/565e578b-d2dc-4fb3-987f-70d19da8148c-bundle\") pod \"565e578b-d2dc-4fb3-987f-70d19da8148c\" (UID: \"565e578b-d2dc-4fb3-987f-70d19da8148c\") " Nov 02 17:33:18.691243 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.688361 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32a6246-abfe-40fa-921d-560e9443aeb0-kube-api-access-9zrz5\" (UniqueName: \"kubernetes.io/projected/d32a6246-abfe-40fa-921d-560e9443aeb0-kube-api-access-9zrz5\") pod \"d32a6246-abfe-40fa-921d-560e9443aeb0\" (UID: \"d32a6246-abfe-40fa-921d-560e9443aeb0\") " Nov 02 17:33:18.691243 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.688430 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32a6246-abfe-40fa-921d-560e9443aeb0-util\" (UniqueName: \"kubernetes.io/empty-dir/d32a6246-abfe-40fa-921d-560e9443aeb0-util\") pod \"d32a6246-abfe-40fa-921d-560e9443aeb0\" (UID: \"d32a6246-abfe-40fa-921d-560e9443aeb0\") " Nov 02 17:33:18.691243 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.688463 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32a6246-abfe-40fa-921d-560e9443aeb0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d32a6246-abfe-40fa-921d-560e9443aeb0-bundle\") pod \"d32a6246-abfe-40fa-921d-560e9443aeb0\" (UID: \"d32a6246-abfe-40fa-921d-560e9443aeb0\") " Nov 02 17:33:18.691243 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.688527 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565e578b-d2dc-4fb3-987f-70d19da8148c-util\" (UniqueName: \"kubernetes.io/empty-dir/565e578b-d2dc-4fb3-987f-70d19da8148c-util\") pod \"565e578b-d2dc-4fb3-987f-70d19da8148c\" (UID: \"565e578b-d2dc-4fb3-987f-70d19da8148c\") " Nov 02 17:33:18.691243 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.688624 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/565e578b-d2dc-4fb3-987f-70d19da8148c-kube-api-access-8rpqg\" (UniqueName: \"kubernetes.io/projected/565e578b-d2dc-4fb3-987f-70d19da8148c-kube-api-access-8rpqg\") pod \"565e578b-d2dc-4fb3-987f-70d19da8148c\" (UID: \"565e578b-d2dc-4fb3-987f-70d19da8148c\") " Nov 02 17:33:18.691243 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.689460 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565e578b-d2dc-4fb3-987f-70d19da8148c-bundle" pod "565e578b-d2dc-4fb3-987f-70d19da8148c" (UID: "565e578b-d2dc-4fb3-987f-70d19da8148c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:18.692351 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.691319 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32a6246-abfe-40fa-921d-560e9443aeb0-bundle" pod "d32a6246-abfe-40fa-921d-560e9443aeb0" (UID: "d32a6246-abfe-40fa-921d-560e9443aeb0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:18.692451 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.692423 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565e578b-d2dc-4fb3-987f-70d19da8148c-kube-api-access-8rpqg" pod "565e578b-d2dc-4fb3-987f-70d19da8148c" (UID: "565e578b-d2dc-4fb3-987f-70d19da8148c"). InnerVolumeSpecName "kube-api-access-8rpqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:33:18.695464 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.695432 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32a6246-abfe-40fa-921d-560e9443aeb0-kube-api-access-9zrz5" pod "d32a6246-abfe-40fa-921d-560e9443aeb0" (UID: "d32a6246-abfe-40fa-921d-560e9443aeb0"). InnerVolumeSpecName "kube-api-access-9zrz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:33:18.696061 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.696011 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565e578b-d2dc-4fb3-987f-70d19da8148c-util" pod "565e578b-d2dc-4fb3-987f-70d19da8148c" (UID: "565e578b-d2dc-4fb3-987f-70d19da8148c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:18.696925 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.696903 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32a6246-abfe-40fa-921d-560e9443aeb0-util" pod "d32a6246-abfe-40fa-921d-560e9443aeb0" (UID: "d32a6246-abfe-40fa-921d-560e9443aeb0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:18.790418 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.790291 4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/565e578b-d2dc-4fb3-987f-70d19da8148c-bundle\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:18.790418 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.790327 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zrz5\" (UniqueName: \"kubernetes.io/projected/d32a6246-abfe-40fa-921d-560e9443aeb0-kube-api-access-9zrz5\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:18.790418 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.790343 4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d32a6246-abfe-40fa-921d-560e9443aeb0-util\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:18.790418 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.790359 4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d32a6246-abfe-40fa-921d-560e9443aeb0-bundle\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:18.790418 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.790373 4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/565e578b-d2dc-4fb3-987f-70d19da8148c-util\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:18.790418 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:18.790386 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rpqg\" (UniqueName: \"kubernetes.io/projected/565e578b-d2dc-4fb3-987f-70d19da8148c-kube-api-access-8rpqg\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:19.213042 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.212969 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7" Nov 02 17:33:19.213042 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.212980 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cptgm7" event={"ID":"565e578b-d2dc-4fb3-987f-70d19da8148c","Type":"ContainerDied","Data":"00d910888423ac9b444d52701c283de2af4745c87e0be63f693e93859d2f7b09"} Nov 02 17:33:19.213042 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.213032 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00d910888423ac9b444d52701c283de2af4745c87e0be63f693e93859d2f7b09" Nov 02 17:33:19.215744 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.215715 4860 generic.go:356] "Generic (PLEG): container finished" podID="3e11a3a4-96e3-42da-905a-4ccb26fde15e" containerID="5ff2ea3bab64070a02f6572dfbacddba5152807f291e5b13783e32c5f8412044" exitCode=0 Nov 02 17:33:19.215914 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.215816 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd" event={"ID":"3e11a3a4-96e3-42da-905a-4ccb26fde15e","Type":"ContainerDied","Data":"5ff2ea3bab64070a02f6572dfbacddba5152807f291e5b13783e32c5f8412044"} Nov 02 17:33:19.217585 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.217561 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 17:33:19.218450 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.218430 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr" Nov 02 17:33:19.218567 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.218448 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr" event={"ID":"4dec5066-dab8-480c-946e-81c8772eacf0","Type":"ContainerDied","Data":"c41e33a0208bb7a1e1ff6d961ed6f3833e4862786d8af96dbdb29807979745b6"} Nov 02 17:33:19.218567 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.218486 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c41e33a0208bb7a1e1ff6d961ed6f3833e4862786d8af96dbdb29807979745b6" Nov 02 17:33:19.222313 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.222045 4860 generic.go:356] "Generic (PLEG): container finished" podID="d633e91f-6217-432f-940e-36ba5fae132e" containerID="b0901e9f0b5c85f945f1da5f8e65b29f1739b6b8a8016f51c6382fe3a9689ee3" exitCode=0 Nov 02 17:33:19.222313 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.222158 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd" event={"ID":"d633e91f-6217-432f-940e-36ba5fae132e","Type":"ContainerDied","Data":"b0901e9f0b5c85f945f1da5f8e65b29f1739b6b8a8016f51c6382fe3a9689ee3"} Nov 02 17:33:19.224891 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.224628 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9" event={"ID":"d32a6246-abfe-40fa-921d-560e9443aeb0","Type":"ContainerDied","Data":"b1803874ef54a6b80b7012cbe94afd7f6955edc3b56dcc36d9abb68654f53b9c"} Nov 02 17:33:19.224891 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.224666 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1803874ef54a6b80b7012cbe94afd7f6955edc3b56dcc36d9abb68654f53b9c" Nov 02 17:33:19.224891 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.224737 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abbwl9" Nov 02 17:33:19.238132 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.238109 4860 generic.go:356] "Generic (PLEG): container finished" podID="7f4595dc-331a-460e-94e8-d3395391c57b" containerID="b5285ab9c41f021c647a327384a8d0193f446385d49499aa9f9cf645789ee7ed" exitCode=0 Nov 02 17:33:19.238290 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.238268 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns" event={"ID":"7f4595dc-331a-460e-94e8-d3395391c57b","Type":"ContainerDied","Data":"b5285ab9c41f021c647a327384a8d0193f446385d49499aa9f9cf645789ee7ed"} Nov 02 17:33:19.414893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.414835 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q" Nov 02 17:33:19.459184 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.459148 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" Nov 02 17:33:19.472058 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.471980 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" Nov 02 17:33:19.496237 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.496205 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c02198c6-d4e4-4bda-90e6-182f01fa068a-kube-api-access-mhrsh\" (UniqueName: \"kubernetes.io/projected/c02198c6-d4e4-4bda-90e6-182f01fa068a-kube-api-access-mhrsh\") pod \"c02198c6-d4e4-4bda-90e6-182f01fa068a\" (UID: \"c02198c6-d4e4-4bda-90e6-182f01fa068a\") " Nov 02 17:33:19.496393 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.496269 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c02198c6-d4e4-4bda-90e6-182f01fa068a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c02198c6-d4e4-4bda-90e6-182f01fa068a-bundle\") pod \"c02198c6-d4e4-4bda-90e6-182f01fa068a\" (UID: \"c02198c6-d4e4-4bda-90e6-182f01fa068a\") " Nov 02 17:33:19.496393 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.496339 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c02198c6-d4e4-4bda-90e6-182f01fa068a-util\" (UniqueName: \"kubernetes.io/empty-dir/c02198c6-d4e4-4bda-90e6-182f01fa068a-util\") pod \"c02198c6-d4e4-4bda-90e6-182f01fa068a\" (UID: \"c02198c6-d4e4-4bda-90e6-182f01fa068a\") " Nov 02 17:33:19.496990 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.496962 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02198c6-d4e4-4bda-90e6-182f01fa068a-bundle" pod "c02198c6-d4e4-4bda-90e6-182f01fa068a" (UID: "c02198c6-d4e4-4bda-90e6-182f01fa068a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:19.498579 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.498548 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02198c6-d4e4-4bda-90e6-182f01fa068a-kube-api-access-mhrsh" pod "c02198c6-d4e4-4bda-90e6-182f01fa068a" (UID: "c02198c6-d4e4-4bda-90e6-182f01fa068a"). InnerVolumeSpecName "kube-api-access-mhrsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:33:19.502321 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.502295 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02198c6-d4e4-4bda-90e6-182f01fa068a-util" pod "c02198c6-d4e4-4bda-90e6-182f01fa068a" (UID: "c02198c6-d4e4-4bda-90e6-182f01fa068a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:19.597482 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.597446 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963e8558-7c81-45fa-96eb-b8e4e11d73f4-kube-api-access-7mlnf\" (UniqueName: \"kubernetes.io/projected/963e8558-7c81-45fa-96eb-b8e4e11d73f4-kube-api-access-7mlnf\") pod \"963e8558-7c81-45fa-96eb-b8e4e11d73f4\" (UID: \"963e8558-7c81-45fa-96eb-b8e4e11d73f4\") " Nov 02 17:33:19.597671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.597523 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963e8558-7c81-45fa-96eb-b8e4e11d73f4-util\" (UniqueName: \"kubernetes.io/empty-dir/963e8558-7c81-45fa-96eb-b8e4e11d73f4-util\") pod \"963e8558-7c81-45fa-96eb-b8e4e11d73f4\" (UID: \"963e8558-7c81-45fa-96eb-b8e4e11d73f4\") " Nov 02 17:33:19.597671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.597614 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963e8558-7c81-45fa-96eb-b8e4e11d73f4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/963e8558-7c81-45fa-96eb-b8e4e11d73f4-bundle\") pod \"963e8558-7c81-45fa-96eb-b8e4e11d73f4\" (UID: \"963e8558-7c81-45fa-96eb-b8e4e11d73f4\") " Nov 02 17:33:19.597795 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.597681 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6891e649-ad88-4bb8-80f3-8f671414df5a-kube-api-access-msgx9\" (UniqueName: \"kubernetes.io/projected/6891e649-ad88-4bb8-80f3-8f671414df5a-kube-api-access-msgx9\") pod \"6891e649-ad88-4bb8-80f3-8f671414df5a\" (UID: \"6891e649-ad88-4bb8-80f3-8f671414df5a\") " Nov 02 17:33:19.597795 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.597718 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6891e649-ad88-4bb8-80f3-8f671414df5a-util\" (UniqueName: \"kubernetes.io/empty-dir/6891e649-ad88-4bb8-80f3-8f671414df5a-util\") pod \"6891e649-ad88-4bb8-80f3-8f671414df5a\" (UID: \"6891e649-ad88-4bb8-80f3-8f671414df5a\") " Nov 02 17:33:19.597952 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.597807 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6891e649-ad88-4bb8-80f3-8f671414df5a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6891e649-ad88-4bb8-80f3-8f671414df5a-bundle\") pod \"6891e649-ad88-4bb8-80f3-8f671414df5a\" (UID: \"6891e649-ad88-4bb8-80f3-8f671414df5a\") " Nov 02 17:33:19.598086 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.598067 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhrsh\" (UniqueName: \"kubernetes.io/projected/c02198c6-d4e4-4bda-90e6-182f01fa068a-kube-api-access-mhrsh\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:19.598148 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.598092 4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c02198c6-d4e4-4bda-90e6-182f01fa068a-bundle\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:19.598148 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.598108 4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c02198c6-d4e4-4bda-90e6-182f01fa068a-util\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:19.598632 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.598579 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963e8558-7c81-45fa-96eb-b8e4e11d73f4-bundle" pod "963e8558-7c81-45fa-96eb-b8e4e11d73f4" (UID: "963e8558-7c81-45fa-96eb-b8e4e11d73f4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:19.598722 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.598659 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6891e649-ad88-4bb8-80f3-8f671414df5a-bundle" pod "6891e649-ad88-4bb8-80f3-8f671414df5a" (UID: "6891e649-ad88-4bb8-80f3-8f671414df5a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:19.600258 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.600232 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963e8558-7c81-45fa-96eb-b8e4e11d73f4-kube-api-access-7mlnf" pod "963e8558-7c81-45fa-96eb-b8e4e11d73f4" (UID: "963e8558-7c81-45fa-96eb-b8e4e11d73f4"). InnerVolumeSpecName "kube-api-access-7mlnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:33:19.600334 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.600299 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6891e649-ad88-4bb8-80f3-8f671414df5a-kube-api-access-msgx9" pod "6891e649-ad88-4bb8-80f3-8f671414df5a" (UID: "6891e649-ad88-4bb8-80f3-8f671414df5a"). InnerVolumeSpecName "kube-api-access-msgx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:33:19.605085 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.605057 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963e8558-7c81-45fa-96eb-b8e4e11d73f4-util" pod "963e8558-7c81-45fa-96eb-b8e4e11d73f4" (UID: "963e8558-7c81-45fa-96eb-b8e4e11d73f4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:19.606696 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.606666 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6891e649-ad88-4bb8-80f3-8f671414df5a-util" pod "6891e649-ad88-4bb8-80f3-8f671414df5a" (UID: "6891e649-ad88-4bb8-80f3-8f671414df5a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:19.699114 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.699075 4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/963e8558-7c81-45fa-96eb-b8e4e11d73f4-util\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:19.699114 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.699110 4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/963e8558-7c81-45fa-96eb-b8e4e11d73f4-bundle\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:19.699114 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.699123 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msgx9\" (UniqueName: \"kubernetes.io/projected/6891e649-ad88-4bb8-80f3-8f671414df5a-kube-api-access-msgx9\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:19.699397 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.699139 4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6891e649-ad88-4bb8-80f3-8f671414df5a-util\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:19.699397 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.699153 4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6891e649-ad88-4bb8-80f3-8f671414df5a-bundle\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:19.699397 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:19.699165 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mlnf\" (UniqueName: \"kubernetes.io/projected/963e8558-7c81-45fa-96eb-b8e4e11d73f4-kube-api-access-7mlnf\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:20.245813 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.245784 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q" Nov 02 17:33:20.246260 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.245778 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7jk6q" event={"ID":"c02198c6-d4e4-4bda-90e6-182f01fa068a","Type":"ContainerDied","Data":"a3d61591f3d9ba4d7e6d026b3e5578ce1dbbe512c2932471ef4419fdada445af"} Nov 02 17:33:20.246260 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.245900 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3d61591f3d9ba4d7e6d026b3e5578ce1dbbe512c2932471ef4419fdada445af" Nov 02 17:33:20.248405 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.248378 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" event={"ID":"d50aed5b-c7b1-4ac2-bb46-d5013752af0e","Type":"ContainerStarted","Data":"c633482a9ee769d526432a97db217b1f5c9911cc56a582686284fca7377e41b4"} Nov 02 17:33:20.250844 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.250819 4860 generic.go:356] "Generic (PLEG): container finished" podID="062ebaf9-1d88-41af-9d5e-5b233f2a1eeb" containerID="4a1b6526b5186f031f1a709845cb53f8510bf5cf3393c40bedb3c1d8a9b513bf" exitCode=0 Nov 02 17:33:20.250992 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.250879 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x" event={"ID":"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb","Type":"ContainerDied","Data":"4a1b6526b5186f031f1a709845cb53f8510bf5cf3393c40bedb3c1d8a9b513bf"} Nov 02 17:33:20.253385 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.253356 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" event={"ID":"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc","Type":"ContainerStarted","Data":"11dc9ca2185b1aa3e1badb2dc2cf9071f50ee4028d5a94b937a54408f125c4ce"} Nov 02 17:33:20.255361 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.255338 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" Nov 02 17:33:20.255470 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.255356 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at66jq" event={"ID":"963e8558-7c81-45fa-96eb-b8e4e11d73f4","Type":"ContainerDied","Data":"e40d4b2012513784eadb1d38c458674d8aa2ec7636d01e5083557a07589abb65"} Nov 02 17:33:20.255470 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.255398 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e40d4b2012513784eadb1d38c458674d8aa2ec7636d01e5083557a07589abb65" Nov 02 17:33:20.257438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.257417 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" Nov 02 17:33:20.257522 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.257448 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nmlqt" event={"ID":"6891e649-ad88-4bb8-80f3-8f671414df5a","Type":"ContainerDied","Data":"96c9ab798efcecdd5373ba8301e1e2a7fa745e9bfd9ee43ce1e525143a4d420e"} Nov 02 17:33:20.257522 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.257491 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96c9ab798efcecdd5373ba8301e1e2a7fa745e9bfd9ee43ce1e525143a4d420e" Nov 02 17:33:20.294881 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.293576 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" podStartSLOduration=2.7655785010000002 podStartE2EDuration="52.293561369s" totalImagesPullingTime="49.527982868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:20.292458151 +0000 UTC m=+218.913817929" watchObservedRunningTime="2026-11-02 17:33:20.293561369 +0000 UTC m=+218.914921146" Nov 02 17:33:20.355678 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.355611 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" podStartSLOduration=3.520867724 podStartE2EDuration="45.355590704s" totalImagesPullingTime="41.83472298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:20.353690868 +0000 UTC m=+218.975050644" watchObservedRunningTime="2026-11-02 17:33:20.355590704 +0000 UTC m=+218.976950480" Nov 02 17:33:20.382188 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.382163 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns" Nov 02 17:33:20.419202 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.419176 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd" Nov 02 17:33:20.461683 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.461656 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd" Nov 02 17:33:20.506268 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.506242 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d633e91f-6217-432f-940e-36ba5fae132e-kube-api-access-khnws\" (UniqueName: \"kubernetes.io/projected/d633e91f-6217-432f-940e-36ba5fae132e-kube-api-access-khnws\") pod \"d633e91f-6217-432f-940e-36ba5fae132e\" (UID: \"d633e91f-6217-432f-940e-36ba5fae132e\") " Nov 02 17:33:20.506392 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.506283 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4595dc-331a-460e-94e8-d3395391c57b-kube-api-access-g44lr\" (UniqueName: \"kubernetes.io/projected/7f4595dc-331a-460e-94e8-d3395391c57b-kube-api-access-g44lr\") pod \"7f4595dc-331a-460e-94e8-d3395391c57b\" (UID: \"7f4595dc-331a-460e-94e8-d3395391c57b\") " Nov 02 17:33:20.506392 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.506322 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4595dc-331a-460e-94e8-d3395391c57b-util\" (UniqueName: \"kubernetes.io/empty-dir/7f4595dc-331a-460e-94e8-d3395391c57b-util\") pod \"7f4595dc-331a-460e-94e8-d3395391c57b\" (UID: \"7f4595dc-331a-460e-94e8-d3395391c57b\") " Nov 02 17:33:20.506392 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.506357 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d633e91f-6217-432f-940e-36ba5fae132e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d633e91f-6217-432f-940e-36ba5fae132e-bundle\") pod \"d633e91f-6217-432f-940e-36ba5fae132e\" (UID: \"d633e91f-6217-432f-940e-36ba5fae132e\") " Nov 02 17:33:20.506524 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.506410 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4595dc-331a-460e-94e8-d3395391c57b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f4595dc-331a-460e-94e8-d3395391c57b-bundle\") pod \"7f4595dc-331a-460e-94e8-d3395391c57b\" (UID: \"7f4595dc-331a-460e-94e8-d3395391c57b\") " Nov 02 17:33:20.506524 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.506494 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d633e91f-6217-432f-940e-36ba5fae132e-util\" (UniqueName: \"kubernetes.io/empty-dir/d633e91f-6217-432f-940e-36ba5fae132e-util\") pod \"d633e91f-6217-432f-940e-36ba5fae132e\" (UID: \"d633e91f-6217-432f-940e-36ba5fae132e\") " Nov 02 17:33:20.507007 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.506979 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4595dc-331a-460e-94e8-d3395391c57b-bundle" pod "7f4595dc-331a-460e-94e8-d3395391c57b" (UID: "7f4595dc-331a-460e-94e8-d3395391c57b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:20.507107 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.507056 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d633e91f-6217-432f-940e-36ba5fae132e-bundle" pod "d633e91f-6217-432f-940e-36ba5fae132e" (UID: "d633e91f-6217-432f-940e-36ba5fae132e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:20.508411 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.508382 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d633e91f-6217-432f-940e-36ba5fae132e-kube-api-access-khnws" pod "d633e91f-6217-432f-940e-36ba5fae132e" (UID: "d633e91f-6217-432f-940e-36ba5fae132e"). InnerVolumeSpecName "kube-api-access-khnws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:33:20.508700 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.508685 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4595dc-331a-460e-94e8-d3395391c57b-kube-api-access-g44lr" pod "7f4595dc-331a-460e-94e8-d3395391c57b" (UID: "7f4595dc-331a-460e-94e8-d3395391c57b"). InnerVolumeSpecName "kube-api-access-g44lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:33:20.514360 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.514324 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4595dc-331a-460e-94e8-d3395391c57b-util" pod "7f4595dc-331a-460e-94e8-d3395391c57b" (UID: "7f4595dc-331a-460e-94e8-d3395391c57b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:20.515596 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.515564 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d633e91f-6217-432f-940e-36ba5fae132e-util" pod "d633e91f-6217-432f-940e-36ba5fae132e" (UID: "d633e91f-6217-432f-940e-36ba5fae132e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:20.607922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.607889 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e11a3a4-96e3-42da-905a-4ccb26fde15e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e11a3a4-96e3-42da-905a-4ccb26fde15e-bundle\") pod \"3e11a3a4-96e3-42da-905a-4ccb26fde15e\" (UID: \"3e11a3a4-96e3-42da-905a-4ccb26fde15e\") " Nov 02 17:33:20.608085 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.607942 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e11a3a4-96e3-42da-905a-4ccb26fde15e-util\" (UniqueName: \"kubernetes.io/empty-dir/3e11a3a4-96e3-42da-905a-4ccb26fde15e-util\") pod \"3e11a3a4-96e3-42da-905a-4ccb26fde15e\" (UID: \"3e11a3a4-96e3-42da-905a-4ccb26fde15e\") " Nov 02 17:33:20.608085 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.608021 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e11a3a4-96e3-42da-905a-4ccb26fde15e-kube-api-access-95psk\" (UniqueName: \"kubernetes.io/projected/3e11a3a4-96e3-42da-905a-4ccb26fde15e-kube-api-access-95psk\") pod \"3e11a3a4-96e3-42da-905a-4ccb26fde15e\" (UID: \"3e11a3a4-96e3-42da-905a-4ccb26fde15e\") " Nov 02 17:33:20.608177 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.608166 4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d633e91f-6217-432f-940e-36ba5fae132e-util\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:20.608208 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.608180 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khnws\" (UniqueName: \"kubernetes.io/projected/d633e91f-6217-432f-940e-36ba5fae132e-kube-api-access-khnws\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:20.608208 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.608190 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g44lr\" (UniqueName: \"kubernetes.io/projected/7f4595dc-331a-460e-94e8-d3395391c57b-kube-api-access-g44lr\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:20.608208 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.608200 4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f4595dc-331a-460e-94e8-d3395391c57b-util\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:20.608299 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.608208 4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d633e91f-6217-432f-940e-36ba5fae132e-bundle\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:20.608299 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.608216 4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f4595dc-331a-460e-94e8-d3395391c57b-bundle\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:20.608487 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.608458 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e11a3a4-96e3-42da-905a-4ccb26fde15e-bundle" pod "3e11a3a4-96e3-42da-905a-4ccb26fde15e" (UID: "3e11a3a4-96e3-42da-905a-4ccb26fde15e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:20.610108 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.610082 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e11a3a4-96e3-42da-905a-4ccb26fde15e-kube-api-access-95psk" pod "3e11a3a4-96e3-42da-905a-4ccb26fde15e" (UID: "3e11a3a4-96e3-42da-905a-4ccb26fde15e"). InnerVolumeSpecName "kube-api-access-95psk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:33:20.614021 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.613988 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e11a3a4-96e3-42da-905a-4ccb26fde15e-util" pod "3e11a3a4-96e3-42da-905a-4ccb26fde15e" (UID: "3e11a3a4-96e3-42da-905a-4ccb26fde15e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:20.709456 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.709416 4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e11a3a4-96e3-42da-905a-4ccb26fde15e-bundle\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:20.709456 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.709446 4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e11a3a4-96e3-42da-905a-4ccb26fde15e-util\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:20.709456 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:20.709457 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95psk\" (UniqueName: \"kubernetes.io/projected/3e11a3a4-96e3-42da-905a-4ccb26fde15e-kube-api-access-95psk\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:21.264508 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.264481 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd" Nov 02 17:33:21.264923 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.264506 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79drcd" event={"ID":"3e11a3a4-96e3-42da-905a-4ccb26fde15e","Type":"ContainerDied","Data":"25be812c5069800d0cb72925b061886c7be3b7577ceb1ec43153483dee37320e"} Nov 02 17:33:21.264923 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.264563 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25be812c5069800d0cb72925b061886c7be3b7577ceb1ec43153483dee37320e" Nov 02 17:33:21.266341 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.266315 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd" event={"ID":"d633e91f-6217-432f-940e-36ba5fae132e","Type":"ContainerDied","Data":"73d147777ded21287e2d07bddccfedc7cdce05b4a450a5405e54044ccce301d8"} Nov 02 17:33:21.266451 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.266357 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73d147777ded21287e2d07bddccfedc7cdce05b4a450a5405e54044ccce301d8" Nov 02 17:33:21.266451 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.266323 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw7hrd" Nov 02 17:33:21.275252 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.275221 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" event={"ID":"d50aed5b-c7b1-4ac2-bb46-d5013752af0e","Type":"ContainerDied","Data":"c633482a9ee769d526432a97db217b1f5c9911cc56a582686284fca7377e41b4"} Nov 02 17:33:21.275668 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.275629 4860 generic.go:356] "Generic (PLEG): container finished" podID="d50aed5b-c7b1-4ac2-bb46-d5013752af0e" containerID="c633482a9ee769d526432a97db217b1f5c9911cc56a582686284fca7377e41b4" exitCode=0 Nov 02 17:33:21.277954 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.277927 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns" Nov 02 17:33:21.278059 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.277924 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62clns" event={"ID":"7f4595dc-331a-460e-94e8-d3395391c57b","Type":"ContainerDied","Data":"6f68ece541630a631f2abc448b1eb5785ba9d37a71ba0e63b746228a7c63d79d"} Nov 02 17:33:21.278059 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.278053 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f68ece541630a631f2abc448b1eb5785ba9d37a71ba0e63b746228a7c63d79d" Nov 02 17:33:21.280397 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.280372 4860 generic.go:356] "Generic (PLEG): container finished" podID="fe8a0285-755a-4758-b3c9-8b93e0a9e8cc" containerID="11dc9ca2185b1aa3e1badb2dc2cf9071f50ee4028d5a94b937a54408f125c4ce" exitCode=0 Nov 02 17:33:21.280478 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.280436 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" event={"ID":"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc","Type":"ContainerDied","Data":"11dc9ca2185b1aa3e1badb2dc2cf9071f50ee4028d5a94b937a54408f125c4ce"} Nov 02 17:33:21.405022 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.404998 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x" Nov 02 17:33:21.517279 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.517186 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-kube-api-access-7c4mn\" (UniqueName: \"kubernetes.io/projected/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-kube-api-access-7c4mn\") pod \"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb\" (UID: \"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb\") " Nov 02 17:33:21.517279 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.517249 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-util\" (UniqueName: \"kubernetes.io/empty-dir/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-util\") pod \"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb\" (UID: \"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb\") " Nov 02 17:33:21.517500 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.517339 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-bundle\") pod \"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb\" (UID: \"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb\") " Nov 02 17:33:21.517713 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.517689 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-bundle" pod "062ebaf9-1d88-41af-9d5e-5b233f2a1eeb" (UID: "062ebaf9-1d88-41af-9d5e-5b233f2a1eeb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:21.519519 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.519485 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-kube-api-access-7c4mn" pod "062ebaf9-1d88-41af-9d5e-5b233f2a1eeb" (UID: "062ebaf9-1d88-41af-9d5e-5b233f2a1eeb"). InnerVolumeSpecName "kube-api-access-7c4mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:33:21.523165 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.523138 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-util" pod "062ebaf9-1d88-41af-9d5e-5b233f2a1eeb" (UID: "062ebaf9-1d88-41af-9d5e-5b233f2a1eeb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:21.619045 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.619002 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c4mn\" (UniqueName: \"kubernetes.io/projected/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-kube-api-access-7c4mn\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:21.619045 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.619040 4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-util\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:21.619045 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:21.619055 4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/062ebaf9-1d88-41af-9d5e-5b233f2a1eeb-bundle\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:21.768192 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:33:21.768113 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:33:22.286976 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.286934 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x" event={"ID":"062ebaf9-1d88-41af-9d5e-5b233f2a1eeb","Type":"ContainerDied","Data":"6d781d50814c287f41988e94a64cdf7ce95998661ff62104d8a5164df6a00480"} Nov 02 17:33:22.286976 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.286979 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442886x" Nov 02 17:33:22.287505 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.286995 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d781d50814c287f41988e94a64cdf7ce95998661ff62104d8a5164df6a00480" Nov 02 17:33:22.421052 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.421029 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" Nov 02 17:33:22.443766 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.443746 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" Nov 02 17:33:22.526674 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.526643 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-kube-api-access-kr9k9\" (UniqueName: \"kubernetes.io/projected/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-kube-api-access-kr9k9\") pod \"d50aed5b-c7b1-4ac2-bb46-d5013752af0e\" (UID: \"d50aed5b-c7b1-4ac2-bb46-d5013752af0e\") " Nov 02 17:33:22.526872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.526720 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-util\" (UniqueName: \"kubernetes.io/empty-dir/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-util\") pod \"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc\" (UID: \"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc\") " Nov 02 17:33:22.526872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.526750 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-util\" (UniqueName: \"kubernetes.io/empty-dir/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-util\") pod \"d50aed5b-c7b1-4ac2-bb46-d5013752af0e\" (UID: \"d50aed5b-c7b1-4ac2-bb46-d5013752af0e\") " Nov 02 17:33:22.526872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.526784 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-bundle\") pod \"d50aed5b-c7b1-4ac2-bb46-d5013752af0e\" (UID: \"d50aed5b-c7b1-4ac2-bb46-d5013752af0e\") " Nov 02 17:33:22.526872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.526824 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-bundle\") pod \"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc\" (UID: \"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc\") " Nov 02 17:33:22.527101 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.526920 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-kube-api-access-6t58f\" (UniqueName: \"kubernetes.io/projected/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-kube-api-access-6t58f\") pod \"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc\" (UID: \"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc\") " Nov 02 17:33:22.527350 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.527325 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-bundle" pod "d50aed5b-c7b1-4ac2-bb46-d5013752af0e" (UID: "d50aed5b-c7b1-4ac2-bb46-d5013752af0e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:22.527480 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.527449 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-bundle" pod "fe8a0285-755a-4758-b3c9-8b93e0a9e8cc" (UID: "fe8a0285-755a-4758-b3c9-8b93e0a9e8cc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:22.527537 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.527513 4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-bundle\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:22.529539 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.529509 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-kube-api-access-6t58f" pod "fe8a0285-755a-4758-b3c9-8b93e0a9e8cc" (UID: "fe8a0285-755a-4758-b3c9-8b93e0a9e8cc"). InnerVolumeSpecName "kube-api-access-6t58f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:33:22.529681 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.529547 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-kube-api-access-kr9k9" pod "d50aed5b-c7b1-4ac2-bb46-d5013752af0e" (UID: "d50aed5b-c7b1-4ac2-bb46-d5013752af0e"). InnerVolumeSpecName "kube-api-access-kr9k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:33:22.532690 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.532659 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-util" pod "d50aed5b-c7b1-4ac2-bb46-d5013752af0e" (UID: "d50aed5b-c7b1-4ac2-bb46-d5013752af0e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:22.535501 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.535471 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-util" pod "fe8a0285-755a-4758-b3c9-8b93e0a9e8cc" (UID: "fe8a0285-755a-4758-b3c9-8b93e0a9e8cc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:22.628822 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.628790 4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-util\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:22.628822 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.628817 4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-util\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:22.628822 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.628826 4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-bundle\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:22.629054 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.628834 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t58f\" (UniqueName: \"kubernetes.io/projected/fe8a0285-755a-4758-b3c9-8b93e0a9e8cc-kube-api-access-6t58f\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:22.629054 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:22.628844 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr9k9\" (UniqueName: \"kubernetes.io/projected/d50aed5b-c7b1-4ac2-bb46-d5013752af0e-kube-api-access-kr9k9\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:23.294405 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:23.294372 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" Nov 02 17:33:23.294405 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:23.294395 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kt5ls" event={"ID":"fe8a0285-755a-4758-b3c9-8b93e0a9e8cc","Type":"ContainerDied","Data":"4283d2f179f24be136c7305c6cfba0c07a954a081b7df4a8db2198bd912088bf"} Nov 02 17:33:23.294927 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:23.294439 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4283d2f179f24be136c7305c6cfba0c07a954a081b7df4a8db2198bd912088bf" Nov 02 17:33:23.296407 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:23.296382 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" event={"ID":"d50aed5b-c7b1-4ac2-bb46-d5013752af0e","Type":"ContainerDied","Data":"36c16573cab6b8e0d4fefbc61324c8f5eb44c5845a474dc81f372dfc6b409e45"} Nov 02 17:33:23.296525 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:23.296415 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36c16573cab6b8e0d4fefbc61324c8f5eb44c5845a474dc81f372dfc6b409e45" Nov 02 17:33:23.296525 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:23.296426 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j6br4" Nov 02 17:33:24.406506 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406471 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2"] Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406785 4860 cpu_manager.go:414] "containerMap: removing container" podUID="fe8a0285-755a-4758-b3c9-8b93e0a9e8cc" containerName="pull" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406796 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe8a0285-755a-4758-b3c9-8b93e0a9e8cc" containerName="pull" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406810 4860 cpu_manager.go:414] "containerMap: removing container" podUID="0caba324-7d75-479c-a1ca-1af2ec3819cb" containerName="insights-gathering" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406815 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0caba324-7d75-479c-a1ca-1af2ec3819cb" containerName="insights-gathering" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406824 4860 cpu_manager.go:414] "containerMap: removing container" podUID="4dec5066-dab8-480c-946e-81c8772eacf0" containerName="util" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406829 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dec5066-dab8-480c-946e-81c8772eacf0" containerName="util" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406837 4860 cpu_manager.go:414] "containerMap: removing container" podUID="963e8558-7c81-45fa-96eb-b8e4e11d73f4" containerName="util" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406841 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963e8558-7c81-45fa-96eb-b8e4e11d73f4" containerName="util" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406859 4860 cpu_manager.go:414] "containerMap: removing container" podUID="062ebaf9-1d88-41af-9d5e-5b233f2a1eeb" containerName="util" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406864 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="062ebaf9-1d88-41af-9d5e-5b233f2a1eeb" containerName="util" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406871 4860 cpu_manager.go:414] "containerMap: removing container" podUID="6891e649-ad88-4bb8-80f3-8f671414df5a" containerName="util" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406875 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6891e649-ad88-4bb8-80f3-8f671414df5a" containerName="util" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406881 4860 cpu_manager.go:414] "containerMap: removing container" podUID="7f4595dc-331a-460e-94e8-d3395391c57b" containerName="pull" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406886 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4595dc-331a-460e-94e8-d3395391c57b" containerName="pull" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406895 4860 cpu_manager.go:414] "containerMap: removing container" podUID="963e8558-7c81-45fa-96eb-b8e4e11d73f4" containerName="pull" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406899 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963e8558-7c81-45fa-96eb-b8e4e11d73f4" containerName="pull" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406916 4860 cpu_manager.go:414] "containerMap: removing container" podUID="d32a6246-abfe-40fa-921d-560e9443aeb0" containerName="util" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406921 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32a6246-abfe-40fa-921d-560e9443aeb0" containerName="util" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406930 4860 cpu_manager.go:414] "containerMap: removing container" podUID="c02198c6-d4e4-4bda-90e6-182f01fa068a" containerName="util" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406935 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c02198c6-d4e4-4bda-90e6-182f01fa068a" containerName="util" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406942 4860 cpu_manager.go:414] "containerMap: removing container" podUID="d50aed5b-c7b1-4ac2-bb46-d5013752af0e" containerName="util" Nov 02 17:33:24.406937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406946 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d50aed5b-c7b1-4ac2-bb46-d5013752af0e" containerName="util" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406951 4860 cpu_manager.go:414] "containerMap: removing container" podUID="3e11a3a4-96e3-42da-905a-4ccb26fde15e" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406956 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e11a3a4-96e3-42da-905a-4ccb26fde15e" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406962 4860 cpu_manager.go:414] "containerMap: removing container" podUID="7f4595dc-331a-460e-94e8-d3395391c57b" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406966 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4595dc-331a-460e-94e8-d3395391c57b" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406974 4860 cpu_manager.go:414] "containerMap: removing container" podUID="3e11a3a4-96e3-42da-905a-4ccb26fde15e" containerName="util" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406978 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e11a3a4-96e3-42da-905a-4ccb26fde15e" containerName="util" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406983 4860 cpu_manager.go:414] "containerMap: removing container" podUID="d50aed5b-c7b1-4ac2-bb46-d5013752af0e" containerName="pull" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406987 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d50aed5b-c7b1-4ac2-bb46-d5013752af0e" containerName="pull" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406993 4860 cpu_manager.go:414] "containerMap: removing container" podUID="963e8558-7c81-45fa-96eb-b8e4e11d73f4" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.406997 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963e8558-7c81-45fa-96eb-b8e4e11d73f4" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407001 4860 cpu_manager.go:414] "containerMap: removing container" podUID="6891e649-ad88-4bb8-80f3-8f671414df5a" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407005 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6891e649-ad88-4bb8-80f3-8f671414df5a" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407014 4860 cpu_manager.go:414] "containerMap: removing container" podUID="7f4595dc-331a-460e-94e8-d3395391c57b" containerName="util" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407019 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4595dc-331a-460e-94e8-d3395391c57b" containerName="util" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407024 4860 cpu_manager.go:414] "containerMap: removing container" podUID="d50aed5b-c7b1-4ac2-bb46-d5013752af0e" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407029 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d50aed5b-c7b1-4ac2-bb46-d5013752af0e" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407039 4860 cpu_manager.go:414] "containerMap: removing container" podUID="565e578b-d2dc-4fb3-987f-70d19da8148c" containerName="util" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407043 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="565e578b-d2dc-4fb3-987f-70d19da8148c" containerName="util" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407049 4860 cpu_manager.go:414] "containerMap: removing container" podUID="4dec5066-dab8-480c-946e-81c8772eacf0" containerName="pull" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407053 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dec5066-dab8-480c-946e-81c8772eacf0" containerName="pull" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407059 4860 cpu_manager.go:414] "containerMap: removing container" podUID="6891e649-ad88-4bb8-80f3-8f671414df5a" containerName="pull" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407063 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6891e649-ad88-4bb8-80f3-8f671414df5a" containerName="pull" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407068 4860 cpu_manager.go:414] "containerMap: removing container" podUID="d633e91f-6217-432f-940e-36ba5fae132e" containerName="pull" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407073 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d633e91f-6217-432f-940e-36ba5fae132e" containerName="pull" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407079 4860 cpu_manager.go:414] "containerMap: removing container" podUID="062ebaf9-1d88-41af-9d5e-5b233f2a1eeb" containerName="pull" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407083 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="062ebaf9-1d88-41af-9d5e-5b233f2a1eeb" containerName="pull" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407088 4860 cpu_manager.go:414] "containerMap: removing container" podUID="565e578b-d2dc-4fb3-987f-70d19da8148c" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407092 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="565e578b-d2dc-4fb3-987f-70d19da8148c" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407099 4860 cpu_manager.go:414] "containerMap: removing container" podUID="fe8a0285-755a-4758-b3c9-8b93e0a9e8cc" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407103 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe8a0285-755a-4758-b3c9-8b93e0a9e8cc" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407108 4860 cpu_manager.go:414] "containerMap: removing container" podUID="3e11a3a4-96e3-42da-905a-4ccb26fde15e" containerName="pull" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407112 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e11a3a4-96e3-42da-905a-4ccb26fde15e" containerName="pull" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407117 4860 cpu_manager.go:414] "containerMap: removing container" podUID="4dec5066-dab8-480c-946e-81c8772eacf0" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407122 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dec5066-dab8-480c-946e-81c8772eacf0" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407129 4860 cpu_manager.go:414] "containerMap: removing container" podUID="c02198c6-d4e4-4bda-90e6-182f01fa068a" containerName="pull" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407133 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c02198c6-d4e4-4bda-90e6-182f01fa068a" containerName="pull" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407139 4860 cpu_manager.go:414] "containerMap: removing container" podUID="565e578b-d2dc-4fb3-987f-70d19da8148c" containerName="pull" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407143 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="565e578b-d2dc-4fb3-987f-70d19da8148c" containerName="pull" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407148 4860 cpu_manager.go:414] "containerMap: removing container" podUID="d32a6246-abfe-40fa-921d-560e9443aeb0" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407153 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32a6246-abfe-40fa-921d-560e9443aeb0" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407158 4860 cpu_manager.go:414] "containerMap: removing container" podUID="c02198c6-d4e4-4bda-90e6-182f01fa068a" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407162 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c02198c6-d4e4-4bda-90e6-182f01fa068a" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407169 4860 cpu_manager.go:414] "containerMap: removing container" podUID="fe8a0285-755a-4758-b3c9-8b93e0a9e8cc" containerName="util" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407173 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe8a0285-755a-4758-b3c9-8b93e0a9e8cc" containerName="util" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407179 4860 cpu_manager.go:414] "containerMap: removing container" podUID="d633e91f-6217-432f-940e-36ba5fae132e" containerName="util" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407185 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d633e91f-6217-432f-940e-36ba5fae132e" containerName="util" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407190 4860 cpu_manager.go:414] "containerMap: removing container" podUID="d633e91f-6217-432f-940e-36ba5fae132e" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407194 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d633e91f-6217-432f-940e-36ba5fae132e" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407199 4860 cpu_manager.go:414] "containerMap: removing container" podUID="062ebaf9-1d88-41af-9d5e-5b233f2a1eeb" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407204 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="062ebaf9-1d88-41af-9d5e-5b233f2a1eeb" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407211 4860 cpu_manager.go:414] "containerMap: removing container" podUID="d32a6246-abfe-40fa-921d-560e9443aeb0" containerName="pull" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407214 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32a6246-abfe-40fa-921d-560e9443aeb0" containerName="pull" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407275 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="963e8558-7c81-45fa-96eb-b8e4e11d73f4" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407283 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6891e649-ad88-4bb8-80f3-8f671414df5a" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407289 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d50aed5b-c7b1-4ac2-bb46-d5013752af0e" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407297 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="565e578b-d2dc-4fb3-987f-70d19da8148c" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407302 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe8a0285-755a-4758-b3c9-8b93e0a9e8cc" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407308 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dec5066-dab8-480c-946e-81c8772eacf0" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407314 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d633e91f-6217-432f-940e-36ba5fae132e" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407319 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32a6246-abfe-40fa-921d-560e9443aeb0" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407324 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c02198c6-d4e4-4bda-90e6-182f01fa068a" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407329 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="062ebaf9-1d88-41af-9d5e-5b233f2a1eeb" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407335 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0caba324-7d75-479c-a1ca-1af2ec3819cb" containerName="insights-gathering" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407341 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e11a3a4-96e3-42da-905a-4ccb26fde15e" containerName="extract" Nov 02 17:33:24.407715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.407345 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4595dc-331a-460e-94e8-d3395391c57b" containerName="extract" Nov 02 17:33:24.409672 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.409467 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:24.443892 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.443834 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 17:33:24.470914 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.470880 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 17:33:24.471089 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.470930 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 17:33:24.471089 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.471037 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 17:33:24.471276 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.471263 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-nv7lh\"" Nov 02 17:33:24.525486 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.525449 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2"] Nov 02 17:33:24.543794 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.543767 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 17:33:24.545097 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.545075 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a6511fbd-6690-4b49-81ef-e34b7dc12857-webhook-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-z5pl2\" (UID: \"a6511fbd-6690-4b49-81ef-e34b7dc12857\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:24.545147 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.545109 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s7qv\" (UniqueName: \"kubernetes.io/projected/a6511fbd-6690-4b49-81ef-e34b7dc12857-kube-api-access-9s7qv\") pod \"opendatahub-operator-controller-manager-76f76b79c-z5pl2\" (UID: \"a6511fbd-6690-4b49-81ef-e34b7dc12857\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:24.545147 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.545139 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6511fbd-6690-4b49-81ef-e34b7dc12857-metrics-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-z5pl2\" (UID: \"a6511fbd-6690-4b49-81ef-e34b7dc12857\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:24.545250 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.545199 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a6511fbd-6690-4b49-81ef-e34b7dc12857-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-z5pl2\" (UID: \"a6511fbd-6690-4b49-81ef-e34b7dc12857\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:24.545329 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.545307 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a6511fbd-6690-4b49-81ef-e34b7dc12857-manifests\") pod \"opendatahub-operator-controller-manager-76f76b79c-z5pl2\" (UID: \"a6511fbd-6690-4b49-81ef-e34b7dc12857\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:24.545390 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.545342 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6511fbd-6690-4b49-81ef-e34b7dc12857-tmp\") pod \"opendatahub-operator-controller-manager-76f76b79c-z5pl2\" (UID: \"a6511fbd-6690-4b49-81ef-e34b7dc12857\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:24.645789 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.645748 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a6511fbd-6690-4b49-81ef-e34b7dc12857-webhook-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-z5pl2\" (UID: \"a6511fbd-6690-4b49-81ef-e34b7dc12857\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:24.645789 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.645792 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s7qv\" (UniqueName: \"kubernetes.io/projected/a6511fbd-6690-4b49-81ef-e34b7dc12857-kube-api-access-9s7qv\") pod \"opendatahub-operator-controller-manager-76f76b79c-z5pl2\" (UID: \"a6511fbd-6690-4b49-81ef-e34b7dc12857\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:24.646103 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.645817 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6511fbd-6690-4b49-81ef-e34b7dc12857-metrics-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-z5pl2\" (UID: \"a6511fbd-6690-4b49-81ef-e34b7dc12857\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:24.646103 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.645833 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a6511fbd-6690-4b49-81ef-e34b7dc12857-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-z5pl2\" (UID: \"a6511fbd-6690-4b49-81ef-e34b7dc12857\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:24.646103 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.645896 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a6511fbd-6690-4b49-81ef-e34b7dc12857-manifests\") pod \"opendatahub-operator-controller-manager-76f76b79c-z5pl2\" (UID: \"a6511fbd-6690-4b49-81ef-e34b7dc12857\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:24.646103 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.645913 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6511fbd-6690-4b49-81ef-e34b7dc12857-tmp\") pod \"opendatahub-operator-controller-manager-76f76b79c-z5pl2\" (UID: \"a6511fbd-6690-4b49-81ef-e34b7dc12857\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:24.646411 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.646377 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a6511fbd-6690-4b49-81ef-e34b7dc12857-manifests\") pod \"opendatahub-operator-controller-manager-76f76b79c-z5pl2\" (UID: \"a6511fbd-6690-4b49-81ef-e34b7dc12857\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:24.648242 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.648217 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6511fbd-6690-4b49-81ef-e34b7dc12857-tmp\") pod \"opendatahub-operator-controller-manager-76f76b79c-z5pl2\" (UID: \"a6511fbd-6690-4b49-81ef-e34b7dc12857\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:24.648466 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.648442 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a6511fbd-6690-4b49-81ef-e34b7dc12857-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-z5pl2\" (UID: \"a6511fbd-6690-4b49-81ef-e34b7dc12857\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:24.648801 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.648780 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a6511fbd-6690-4b49-81ef-e34b7dc12857-webhook-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-z5pl2\" (UID: \"a6511fbd-6690-4b49-81ef-e34b7dc12857\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:24.648839 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.648783 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6511fbd-6690-4b49-81ef-e34b7dc12857-metrics-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-z5pl2\" (UID: \"a6511fbd-6690-4b49-81ef-e34b7dc12857\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:24.670258 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.670201 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s7qv\" (UniqueName: \"kubernetes.io/projected/a6511fbd-6690-4b49-81ef-e34b7dc12857-kube-api-access-9s7qv\") pod \"opendatahub-operator-controller-manager-76f76b79c-z5pl2\" (UID: \"a6511fbd-6690-4b49-81ef-e34b7dc12857\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:24.721807 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:24.721762 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:25.077031 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:25.075644 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2"] Nov 02 17:33:25.078039 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:33:25.078005 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6511fbd_6690_4b49_81ef_e34b7dc12857.slice/crio-0c170512d26ec5e4dca1854a7b88953ff2e4a6863d77ddace0f4f776a0529408 WatchSource:0}: Error finding container 0c170512d26ec5e4dca1854a7b88953ff2e4a6863d77ddace0f4f776a0529408: Status 404 returned error can't find the container with id 0c170512d26ec5e4dca1854a7b88953ff2e4a6863d77ddace0f4f776a0529408 Nov 02 17:33:25.307445 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:25.307409 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" event={"ID":"a6511fbd-6690-4b49-81ef-e34b7dc12857","Type":"ContainerStarted","Data":"0c170512d26ec5e4dca1854a7b88953ff2e4a6863d77ddace0f4f776a0529408"} Nov 02 17:33:26.038961 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:26.038913 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm"] Nov 02 17:33:26.041673 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:26.041648 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm" Nov 02 17:33:26.060229 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:26.060185 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 17:33:26.073525 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:26.072752 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 17:33:26.073525 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:26.073495 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-fztfg\"" Nov 02 17:33:26.076012 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:26.075982 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm"] Nov 02 17:33:26.158313 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:26.157944 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm\" (UID: \"129ab4d6-054b-44f8-b8fb-3ca138f49f0a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm" Nov 02 17:33:26.158313 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:26.158111 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rlcx\" (UniqueName: \"kubernetes.io/projected/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-kube-api-access-2rlcx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm\" (UID: \"129ab4d6-054b-44f8-b8fb-3ca138f49f0a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm" Nov 02 17:33:26.158313 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:26.158153 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm\" (UID: \"129ab4d6-054b-44f8-b8fb-3ca138f49f0a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm" Nov 02 17:33:26.259289 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:26.259246 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlcx\" (UniqueName: \"kubernetes.io/projected/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-kube-api-access-2rlcx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm\" (UID: \"129ab4d6-054b-44f8-b8fb-3ca138f49f0a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm" Nov 02 17:33:26.259437 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:26.259299 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm\" (UID: \"129ab4d6-054b-44f8-b8fb-3ca138f49f0a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm" Nov 02 17:33:26.259437 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:26.259333 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm\" (UID: \"129ab4d6-054b-44f8-b8fb-3ca138f49f0a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm" Nov 02 17:33:26.259742 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:26.259724 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm\" (UID: \"129ab4d6-054b-44f8-b8fb-3ca138f49f0a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm" Nov 02 17:33:26.260251 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:26.260225 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm\" (UID: \"129ab4d6-054b-44f8-b8fb-3ca138f49f0a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm" Nov 02 17:33:26.289340 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:26.289268 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rlcx\" (UniqueName: \"kubernetes.io/projected/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-kube-api-access-2rlcx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm\" (UID: \"129ab4d6-054b-44f8-b8fb-3ca138f49f0a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm" Nov 02 17:33:26.358597 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:26.358564 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm" Nov 02 17:33:26.674377 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:26.674347 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm"] Nov 02 17:33:26.676256 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:33:26.676228 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129ab4d6_054b_44f8_b8fb_3ca138f49f0a.slice/crio-d4a392a88c39cdf0bd2f5045e9b6debcac42c26de5c5247e5a5d2d92a908cb4d WatchSource:0}: Error finding container d4a392a88c39cdf0bd2f5045e9b6debcac42c26de5c5247e5a5d2d92a908cb4d: Status 404 returned error can't find the container with id d4a392a88c39cdf0bd2f5045e9b6debcac42c26de5c5247e5a5d2d92a908cb4d Nov 02 17:33:27.323919 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:27.323878 4860 generic.go:356] "Generic (PLEG): container finished" podID="129ab4d6-054b-44f8-b8fb-3ca138f49f0a" containerID="85f37483ea70bd510b2ac012fa5756bc58c645599f2f90633b29bc3e18cc4d51" exitCode=0 Nov 02 17:33:27.324345 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:27.323942 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm" event={"ID":"129ab4d6-054b-44f8-b8fb-3ca138f49f0a","Type":"ContainerDied","Data":"85f37483ea70bd510b2ac012fa5756bc58c645599f2f90633b29bc3e18cc4d51"} Nov 02 17:33:27.324345 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:27.324004 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm" event={"ID":"129ab4d6-054b-44f8-b8fb-3ca138f49f0a","Type":"ContainerStarted","Data":"d4a392a88c39cdf0bd2f5045e9b6debcac42c26de5c5247e5a5d2d92a908cb4d"} Nov 02 17:33:28.330462 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:28.330418 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" event={"ID":"a6511fbd-6690-4b49-81ef-e34b7dc12857","Type":"ContainerStarted","Data":"bd4694b0b19d43d0ceda81077057a70984fae05a14cca08850d108b88610ac38"} Nov 02 17:33:28.332435 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:28.332409 4860 generic.go:356] "Generic (PLEG): container finished" podID="129ab4d6-054b-44f8-b8fb-3ca138f49f0a" containerID="354c33f11fc8ad1f407278fe51ee664f3e32390e4c91d2ffd85fc6deda54c263" exitCode=0 Nov 02 17:33:28.332575 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:28.332473 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm" event={"ID":"129ab4d6-054b-44f8-b8fb-3ca138f49f0a","Type":"ContainerDied","Data":"354c33f11fc8ad1f407278fe51ee664f3e32390e4c91d2ffd85fc6deda54c263"} Nov 02 17:33:29.339867 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:29.339812 4860 generic.go:356] "Generic (PLEG): container finished" podID="129ab4d6-054b-44f8-b8fb-3ca138f49f0a" containerID="8f72d1466ebcfa485f75066cf953d00b387fd7533e27a77839dd7bd8353a231e" exitCode=0 Nov 02 17:33:29.340332 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:29.339886 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm" event={"ID":"129ab4d6-054b-44f8-b8fb-3ca138f49f0a","Type":"ContainerDied","Data":"8f72d1466ebcfa485f75066cf953d00b387fd7533e27a77839dd7bd8353a231e"} Nov 02 17:33:29.341522 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:29.341496 4860 generic.go:356] "Generic (PLEG): container finished" podID="a6511fbd-6690-4b49-81ef-e34b7dc12857" containerID="bd4694b0b19d43d0ceda81077057a70984fae05a14cca08850d108b88610ac38" exitCode=0 Nov 02 17:33:29.341655 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:29.341552 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" event={"ID":"a6511fbd-6690-4b49-81ef-e34b7dc12857","Type":"ContainerDied","Data":"bd4694b0b19d43d0ceda81077057a70984fae05a14cca08850d108b88610ac38"} Nov 02 17:33:29.877678 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:29.877641 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-jl7g4"] Nov 02 17:33:29.879950 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:29.879926 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jl7g4" Nov 02 17:33:29.900189 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:29.900130 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-dl429\"" Nov 02 17:33:29.903709 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:29.903687 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-jl7g4"] Nov 02 17:33:29.944644 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:29.944618 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:33:29.944644 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:29.944648 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 17:33:29.990025 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:29.989992 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5q59\" (UniqueName: \"kubernetes.io/projected/5a37c6dc-7a4b-4861-b2ef-9b6d8d0543d3-kube-api-access-c5q59\") pod \"jobset-operator-674cd59dc5-jl7g4\" (UID: \"5a37c6dc-7a4b-4861-b2ef-9b6d8d0543d3\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jl7g4" Nov 02 17:33:29.990148 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:29.990122 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a37c6dc-7a4b-4861-b2ef-9b6d8d0543d3-tmp\") pod \"jobset-operator-674cd59dc5-jl7g4\" (UID: \"5a37c6dc-7a4b-4861-b2ef-9b6d8d0543d3\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jl7g4" Nov 02 17:33:30.091291 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:30.091258 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a37c6dc-7a4b-4861-b2ef-9b6d8d0543d3-tmp\") pod \"jobset-operator-674cd59dc5-jl7g4\" (UID: \"5a37c6dc-7a4b-4861-b2ef-9b6d8d0543d3\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jl7g4" Nov 02 17:33:30.091438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:30.091298 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5q59\" (UniqueName: \"kubernetes.io/projected/5a37c6dc-7a4b-4861-b2ef-9b6d8d0543d3-kube-api-access-c5q59\") pod \"jobset-operator-674cd59dc5-jl7g4\" (UID: \"5a37c6dc-7a4b-4861-b2ef-9b6d8d0543d3\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jl7g4" Nov 02 17:33:30.091658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:30.091633 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a37c6dc-7a4b-4861-b2ef-9b6d8d0543d3-tmp\") pod \"jobset-operator-674cd59dc5-jl7g4\" (UID: \"5a37c6dc-7a4b-4861-b2ef-9b6d8d0543d3\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jl7g4" Nov 02 17:33:30.115427 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:30.115394 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5q59\" (UniqueName: \"kubernetes.io/projected/5a37c6dc-7a4b-4861-b2ef-9b6d8d0543d3-kube-api-access-c5q59\") pod \"jobset-operator-674cd59dc5-jl7g4\" (UID: \"5a37c6dc-7a4b-4861-b2ef-9b6d8d0543d3\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jl7g4" Nov 02 17:33:30.203693 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:30.203620 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jl7g4" Nov 02 17:33:30.348488 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:30.348452 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" event={"ID":"a6511fbd-6690-4b49-81ef-e34b7dc12857","Type":"ContainerStarted","Data":"229d493651b285711f4d328da129705b1c6bb0ba96d94a01fab519b415e4da6c"} Nov 02 17:33:30.348922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:30.348537 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:30.423452 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:30.423396 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" podStartSLOduration=2.7033577170000003 podStartE2EDuration="6.423381328s" totalImagesPullingTime="3.22120673s" totalInitContainerRuntime="498.816881ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:30.42226732 +0000 UTC m=+229.043627096" watchObservedRunningTime="2026-11-02 17:33:30.423381328 +0000 UTC m=+229.044741104" Nov 02 17:33:30.469125 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:30.469101 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm" Nov 02 17:33:30.513789 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:30.513770 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-jl7g4"] Nov 02 17:33:30.515330 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:33:30.515303 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a37c6dc_7a4b_4861_b2ef_9b6d8d0543d3.slice/crio-bcd06165291d4eda20d95b943fddec0fd0a615f27312550c464c8490826d4fdd WatchSource:0}: Error finding container bcd06165291d4eda20d95b943fddec0fd0a615f27312550c464c8490826d4fdd: Status 404 returned error can't find the container with id bcd06165291d4eda20d95b943fddec0fd0a615f27312550c464c8490826d4fdd Nov 02 17:33:30.595415 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:30.595381 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-kube-api-access-2rlcx\" (UniqueName: \"kubernetes.io/projected/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-kube-api-access-2rlcx\") pod \"129ab4d6-054b-44f8-b8fb-3ca138f49f0a\" (UID: \"129ab4d6-054b-44f8-b8fb-3ca138f49f0a\") " Nov 02 17:33:30.595550 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:30.595458 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-util\" (UniqueName: \"kubernetes.io/empty-dir/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-util\") pod \"129ab4d6-054b-44f8-b8fb-3ca138f49f0a\" (UID: \"129ab4d6-054b-44f8-b8fb-3ca138f49f0a\") " Nov 02 17:33:30.595550 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:30.595481 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-bundle\") pod \"129ab4d6-054b-44f8-b8fb-3ca138f49f0a\" (UID: \"129ab4d6-054b-44f8-b8fb-3ca138f49f0a\") " Nov 02 17:33:30.596202 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:30.596168 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-bundle" pod "129ab4d6-054b-44f8-b8fb-3ca138f49f0a" (UID: "129ab4d6-054b-44f8-b8fb-3ca138f49f0a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:30.597440 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:30.597418 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-kube-api-access-2rlcx" pod "129ab4d6-054b-44f8-b8fb-3ca138f49f0a" (UID: "129ab4d6-054b-44f8-b8fb-3ca138f49f0a"). InnerVolumeSpecName "kube-api-access-2rlcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:33:30.601379 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:30.601354 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-util" pod "129ab4d6-054b-44f8-b8fb-3ca138f49f0a" (UID: "129ab4d6-054b-44f8-b8fb-3ca138f49f0a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:33:30.696574 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:30.696541 4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-util\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:30.696574 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:30.696570 4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-bundle\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:30.696728 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:30.696580 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rlcx\" (UniqueName: \"kubernetes.io/projected/129ab4d6-054b-44f8-b8fb-3ca138f49f0a-kube-api-access-2rlcx\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:33:31.354661 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:31.354622 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jl7g4" event={"ID":"5a37c6dc-7a4b-4861-b2ef-9b6d8d0543d3","Type":"ContainerStarted","Data":"bcd06165291d4eda20d95b943fddec0fd0a615f27312550c464c8490826d4fdd"} Nov 02 17:33:31.356502 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:31.356469 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm" event={"ID":"129ab4d6-054b-44f8-b8fb-3ca138f49f0a","Type":"ContainerDied","Data":"d4a392a88c39cdf0bd2f5045e9b6debcac42c26de5c5247e5a5d2d92a908cb4d"} Nov 02 17:33:31.356601 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:31.356509 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7s9xm" Nov 02 17:33:31.356601 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:31.356526 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4a392a88c39cdf0bd2f5045e9b6debcac42c26de5c5247e5a5d2d92a908cb4d" Nov 02 17:33:33.366932 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:33.366883 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jl7g4" event={"ID":"5a37c6dc-7a4b-4861-b2ef-9b6d8d0543d3","Type":"ContainerStarted","Data":"5fd8e2fcec9fe29cb408c156073fa682275d8618fa404f897c8f7c54513bd8be"} Nov 02 17:33:33.423154 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:33.423090 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jl7g4" podStartSLOduration=2.088836083 podStartE2EDuration="4.423075191s" totalImagesPullingTime="2.334239108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:33.422422838 +0000 UTC m=+232.043782613" watchObservedRunningTime="2026-11-02 17:33:33.423075191 +0000 UTC m=+232.044434967" Nov 02 17:33:41.359810 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:41.359777 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-z5pl2" Nov 02 17:33:41.854484 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:41.854452 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-thmg4"] Nov 02 17:33:41.854765 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:41.854753 4860 cpu_manager.go:414] "containerMap: removing container" podUID="129ab4d6-054b-44f8-b8fb-3ca138f49f0a" containerName="extract" Nov 02 17:33:41.854806 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:41.854766 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129ab4d6-054b-44f8-b8fb-3ca138f49f0a" containerName="extract" Nov 02 17:33:41.854806 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:41.854790 4860 cpu_manager.go:414] "containerMap: removing container" podUID="129ab4d6-054b-44f8-b8fb-3ca138f49f0a" containerName="pull" Nov 02 17:33:41.854806 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:41.854796 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129ab4d6-054b-44f8-b8fb-3ca138f49f0a" containerName="pull" Nov 02 17:33:41.854806 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:41.854804 4860 cpu_manager.go:414] "containerMap: removing container" podUID="129ab4d6-054b-44f8-b8fb-3ca138f49f0a" containerName="util" Nov 02 17:33:41.854959 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:41.854809 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129ab4d6-054b-44f8-b8fb-3ca138f49f0a" containerName="util" Nov 02 17:33:41.854959 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:41.854888 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="129ab4d6-054b-44f8-b8fb-3ca138f49f0a" containerName="extract" Nov 02 17:33:41.903478 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:41.903451 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-thmg4"] Nov 02 17:33:41.903624 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:41.903547 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thmg4" Nov 02 17:33:41.921070 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:41.921027 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-d9x5b\"" Nov 02 17:33:41.921226 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:41.921214 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 17:33:41.921483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:41.921471 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 17:33:41.989576 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:41.989540 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8hsp\" (UniqueName: \"kubernetes.io/projected/995df54a-11d6-4015-9c3a-431d142cfba0-kube-api-access-q8hsp\") pod \"limitador-operator-controller-manager-5778bb4449-thmg4\" (UID: \"995df54a-11d6-4015-9c3a-431d142cfba0\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thmg4" Nov 02 17:33:42.090248 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:42.090209 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8hsp\" (UniqueName: \"kubernetes.io/projected/995df54a-11d6-4015-9c3a-431d142cfba0-kube-api-access-q8hsp\") pod \"limitador-operator-controller-manager-5778bb4449-thmg4\" (UID: \"995df54a-11d6-4015-9c3a-431d142cfba0\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thmg4" Nov 02 17:33:42.123812 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:42.123751 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8hsp\" (UniqueName: \"kubernetes.io/projected/995df54a-11d6-4015-9c3a-431d142cfba0-kube-api-access-q8hsp\") pod \"limitador-operator-controller-manager-5778bb4449-thmg4\" (UID: \"995df54a-11d6-4015-9c3a-431d142cfba0\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thmg4" Nov 02 17:33:42.232277 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:42.232243 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thmg4" Nov 02 17:33:42.540377 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:42.540353 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-thmg4"] Nov 02 17:33:42.542004 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:33:42.541967 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995df54a_11d6_4015_9c3a_431d142cfba0.slice/crio-147a64936f574d262ae11b40c66d4dba7d88527bb01c900417e0a4b4c5d64ce1 WatchSource:0}: Error finding container 147a64936f574d262ae11b40c66d4dba7d88527bb01c900417e0a4b4c5d64ce1: Status 404 returned error can't find the container with id 147a64936f574d262ae11b40c66d4dba7d88527bb01c900417e0a4b4c5d64ce1 Nov 02 17:33:43.425662 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:43.425621 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thmg4" event={"ID":"995df54a-11d6-4015-9c3a-431d142cfba0","Type":"ContainerStarted","Data":"147a64936f574d262ae11b40c66d4dba7d88527bb01c900417e0a4b4c5d64ce1"} Nov 02 17:33:45.438057 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:45.438020 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thmg4" event={"ID":"995df54a-11d6-4015-9c3a-431d142cfba0","Type":"ContainerStarted","Data":"80cb777965906612694c3951dd9d3aaff4e8699471ab691be8a1385de2ebeb23"} Nov 02 17:33:45.438465 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:45.438084 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thmg4" Nov 02 17:33:45.483969 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:45.483925 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thmg4" podStartSLOduration=2.097673632 podStartE2EDuration="4.483913254s" totalImagesPullingTime="2.386239622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:45.483697237 +0000 UTC m=+244.105057014" watchObservedRunningTime="2026-11-02 17:33:45.483913254 +0000 UTC m=+244.105273029" Nov 02 17:33:48.376511 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:48.376473 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b"] Nov 02 17:33:48.381650 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:48.381630 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b" Nov 02 17:33:48.437052 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:48.437012 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:33:48.437218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:48.437165 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 02 17:33:48.437294 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:48.437246 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-6l765\"" Nov 02 17:33:48.440187 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:48.440163 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b"] Nov 02 17:33:48.447616 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:48.447593 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr8bk\" (UniqueName: \"kubernetes.io/projected/8f10d062-8823-4ebc-bbd2-6d04957bdea5-kube-api-access-wr8bk\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b\" (UID: \"8f10d062-8823-4ebc-bbd2-6d04957bdea5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b" Nov 02 17:33:48.447679 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:48.447663 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f10d062-8823-4ebc-bbd2-6d04957bdea5-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b\" (UID: \"8f10d062-8823-4ebc-bbd2-6d04957bdea5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b" Nov 02 17:33:48.548539 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:48.548506 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr8bk\" (UniqueName: \"kubernetes.io/projected/8f10d062-8823-4ebc-bbd2-6d04957bdea5-kube-api-access-wr8bk\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b\" (UID: \"8f10d062-8823-4ebc-bbd2-6d04957bdea5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b" Nov 02 17:33:48.548693 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:48.548576 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f10d062-8823-4ebc-bbd2-6d04957bdea5-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b\" (UID: \"8f10d062-8823-4ebc-bbd2-6d04957bdea5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b" Nov 02 17:33:48.548938 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:48.548921 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f10d062-8823-4ebc-bbd2-6d04957bdea5-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b\" (UID: \"8f10d062-8823-4ebc-bbd2-6d04957bdea5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b" Nov 02 17:33:48.577476 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:48.577448 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr8bk\" (UniqueName: \"kubernetes.io/projected/8f10d062-8823-4ebc-bbd2-6d04957bdea5-kube-api-access-wr8bk\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b\" (UID: \"8f10d062-8823-4ebc-bbd2-6d04957bdea5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b" Nov 02 17:33:48.695329 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:48.695240 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b" Nov 02 17:33:49.031677 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:49.031600 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b"] Nov 02 17:33:49.044975 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:33:49.044943 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f10d062_8823_4ebc_bbd2_6d04957bdea5.slice/crio-9a2f8b0fe939fdf569ca3ca045671c327f118eff89ac2ace41e8b6e2d7e4a37f WatchSource:0}: Error finding container 9a2f8b0fe939fdf569ca3ca045671c327f118eff89ac2ace41e8b6e2d7e4a37f: Status 404 returned error can't find the container with id 9a2f8b0fe939fdf569ca3ca045671c327f118eff89ac2ace41e8b6e2d7e4a37f Nov 02 17:33:49.465971 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:49.465936 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b" event={"ID":"8f10d062-8823-4ebc-bbd2-6d04957bdea5","Type":"ContainerStarted","Data":"9a2f8b0fe939fdf569ca3ca045671c327f118eff89ac2ace41e8b6e2d7e4a37f"} Nov 02 17:33:49.572173 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:49.572139 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj"] Nov 02 17:33:49.577903 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:49.577883 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj" Nov 02 17:33:49.599013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:49.598969 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p2jm6\"" Nov 02 17:33:49.601719 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:49.601692 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj"] Nov 02 17:33:49.657305 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:49.657271 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a049234f-a867-41f5-969f-5ac24f6bfb13-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rs7mj\" (UID: \"a049234f-a867-41f5-969f-5ac24f6bfb13\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj" Nov 02 17:33:49.657480 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:49.657391 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxtb8\" (UniqueName: \"kubernetes.io/projected/a049234f-a867-41f5-969f-5ac24f6bfb13-kube-api-access-nxtb8\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rs7mj\" (UID: \"a049234f-a867-41f5-969f-5ac24f6bfb13\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj" Nov 02 17:33:49.758186 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:49.758104 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a049234f-a867-41f5-969f-5ac24f6bfb13-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rs7mj\" (UID: \"a049234f-a867-41f5-969f-5ac24f6bfb13\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj" Nov 02 17:33:49.758186 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:49.758178 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxtb8\" (UniqueName: \"kubernetes.io/projected/a049234f-a867-41f5-969f-5ac24f6bfb13-kube-api-access-nxtb8\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rs7mj\" (UID: \"a049234f-a867-41f5-969f-5ac24f6bfb13\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj" Nov 02 17:33:49.758595 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:49.758560 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a049234f-a867-41f5-969f-5ac24f6bfb13-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rs7mj\" (UID: \"a049234f-a867-41f5-969f-5ac24f6bfb13\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj" Nov 02 17:33:49.784785 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:49.784756 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxtb8\" (UniqueName: \"kubernetes.io/projected/a049234f-a867-41f5-969f-5ac24f6bfb13-kube-api-access-nxtb8\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rs7mj\" (UID: \"a049234f-a867-41f5-969f-5ac24f6bfb13\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj" Nov 02 17:33:49.890975 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:49.890940 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj" Nov 02 17:33:50.242453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:50.242424 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj"] Nov 02 17:33:50.244785 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:33:50.244752 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda049234f_a867_41f5_969f_5ac24f6bfb13.slice/crio-55f9702619aedea92682145c50f33e80be2e7b6d8dc0d2d8b6aa7333518119ed WatchSource:0}: Error finding container 55f9702619aedea92682145c50f33e80be2e7b6d8dc0d2d8b6aa7333518119ed: Status 404 returned error can't find the container with id 55f9702619aedea92682145c50f33e80be2e7b6d8dc0d2d8b6aa7333518119ed Nov 02 17:33:50.473695 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:50.473658 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj" event={"ID":"a049234f-a867-41f5-969f-5ac24f6bfb13","Type":"ContainerStarted","Data":"55f9702619aedea92682145c50f33e80be2e7b6d8dc0d2d8b6aa7333518119ed"} Nov 02 17:33:51.483223 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:51.483188 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b" event={"ID":"8f10d062-8823-4ebc-bbd2-6d04957bdea5","Type":"ContainerStarted","Data":"1aa7c10e546b7651aca5310987111280a65d1c3890aea8a09ed3d977d1fdea0b"} Nov 02 17:33:51.556329 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:51.556269 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ftx4b" podStartSLOduration=1.973672891 podStartE2EDuration="3.55625098s" totalImagesPullingTime="1.582578089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:51.542601921 +0000 UTC m=+250.163961698" watchObservedRunningTime="2026-11-02 17:33:51.55625098 +0000 UTC m=+250.177610758" Nov 02 17:33:51.775291 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:33:51.775206 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:33:54.468383 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:54.468305 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-sjclp"] Nov 02 17:33:54.471820 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:54.471798 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-sjclp" Nov 02 17:33:54.501065 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:54.501037 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km7dd\" (UniqueName: \"kubernetes.io/projected/48bcace2-94a5-4c4a-b7e7-ee2cb72dedfc-kube-api-access-km7dd\") pod \"authorino-operator-7b7bb4bc77-sjclp\" (UID: \"48bcace2-94a5-4c4a-b7e7-ee2cb72dedfc\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-sjclp" Nov 02 17:33:54.502087 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:54.502059 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj" event={"ID":"a049234f-a867-41f5-969f-5ac24f6bfb13","Type":"ContainerStarted","Data":"7b2de1a3482036a5b787d26ecdb6eca907ea8f514f8a477512569fd61fbfb30e"} Nov 02 17:33:54.502221 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:54.502128 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj" Nov 02 17:33:54.515434 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:54.515412 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-gb97l\"" Nov 02 17:33:54.517491 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:54.517467 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-sjclp"] Nov 02 17:33:54.597458 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:54.597406 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj" podStartSLOduration=1.7271502349999999 podStartE2EDuration="5.597392771s" totalImagesPullingTime="3.870242536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:54.594976612 +0000 UTC m=+253.216336387" watchObservedRunningTime="2026-11-02 17:33:54.597392771 +0000 UTC m=+253.218752547" Nov 02 17:33:54.602007 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:54.601980 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km7dd\" (UniqueName: \"kubernetes.io/projected/48bcace2-94a5-4c4a-b7e7-ee2cb72dedfc-kube-api-access-km7dd\") pod \"authorino-operator-7b7bb4bc77-sjclp\" (UID: \"48bcace2-94a5-4c4a-b7e7-ee2cb72dedfc\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-sjclp" Nov 02 17:33:54.637693 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:54.637662 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km7dd\" (UniqueName: \"kubernetes.io/projected/48bcace2-94a5-4c4a-b7e7-ee2cb72dedfc-kube-api-access-km7dd\") pod \"authorino-operator-7b7bb4bc77-sjclp\" (UID: \"48bcace2-94a5-4c4a-b7e7-ee2cb72dedfc\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-sjclp" Nov 02 17:33:54.785658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:54.785577 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-sjclp" Nov 02 17:33:55.088203 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:55.088175 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-sjclp"] Nov 02 17:33:55.089586 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:33:55.089558 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bcace2_94a5_4c4a_b7e7_ee2cb72dedfc.slice/crio-8986412231bfa74b65567311ed16199af11eb7975b0845c91645f985bf24282b WatchSource:0}: Error finding container 8986412231bfa74b65567311ed16199af11eb7975b0845c91645f985bf24282b: Status 404 returned error can't find the container with id 8986412231bfa74b65567311ed16199af11eb7975b0845c91645f985bf24282b Nov 02 17:33:55.507616 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:55.507540 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-sjclp" event={"ID":"48bcace2-94a5-4c4a-b7e7-ee2cb72dedfc","Type":"ContainerStarted","Data":"8986412231bfa74b65567311ed16199af11eb7975b0845c91645f985bf24282b"} Nov 02 17:33:56.446038 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:56.446003 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-thmg4" Nov 02 17:33:58.083622 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:58.083563 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-d6j7v"] Nov 02 17:33:58.087176 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:58.087156 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d6j7v" Nov 02 17:33:58.120608 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:58.120579 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7m7gg\"" Nov 02 17:33:58.120608 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:58.120579 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 02 17:33:58.123065 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:58.123037 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-d6j7v"] Nov 02 17:33:58.131095 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:58.131073 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glhb9\" (UniqueName: \"kubernetes.io/projected/d82bf74d-62d0-415d-aa93-abce6ae44ba7-kube-api-access-glhb9\") pod \"dns-operator-controller-manager-8557cbdb84-d6j7v\" (UID: \"d82bf74d-62d0-415d-aa93-abce6ae44ba7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d6j7v" Nov 02 17:33:58.231515 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:58.231483 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glhb9\" (UniqueName: \"kubernetes.io/projected/d82bf74d-62d0-415d-aa93-abce6ae44ba7-kube-api-access-glhb9\") pod \"dns-operator-controller-manager-8557cbdb84-d6j7v\" (UID: \"d82bf74d-62d0-415d-aa93-abce6ae44ba7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d6j7v" Nov 02 17:33:58.257410 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:58.257379 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glhb9\" (UniqueName: \"kubernetes.io/projected/d82bf74d-62d0-415d-aa93-abce6ae44ba7-kube-api-access-glhb9\") pod \"dns-operator-controller-manager-8557cbdb84-d6j7v\" (UID: \"d82bf74d-62d0-415d-aa93-abce6ae44ba7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d6j7v" Nov 02 17:33:58.401602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:58.401514 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d6j7v" Nov 02 17:33:58.741948 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:58.741924 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-d6j7v"] Nov 02 17:33:58.743837 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:33:58.743809 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82bf74d_62d0_415d_aa93_abce6ae44ba7.slice/crio-048e331ac8651337e1834d2bce97ccdc44cc24fc875e19bc4f8640a72e3b89d1 WatchSource:0}: Error finding container 048e331ac8651337e1834d2bce97ccdc44cc24fc875e19bc4f8640a72e3b89d1: Status 404 returned error can't find the container with id 048e331ac8651337e1834d2bce97ccdc44cc24fc875e19bc4f8640a72e3b89d1 Nov 02 17:33:59.531216 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:59.531177 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-sjclp" event={"ID":"48bcace2-94a5-4c4a-b7e7-ee2cb72dedfc","Type":"ContainerStarted","Data":"3c00a95e1a6d2cdc572eca364a3ba6b185d0ef32903667be25f91b39a852a469"} Nov 02 17:33:59.531646 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:59.531330 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-sjclp" Nov 02 17:33:59.532721 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:59.532695 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d6j7v" event={"ID":"d82bf74d-62d0-415d-aa93-abce6ae44ba7","Type":"ContainerStarted","Data":"048e331ac8651337e1834d2bce97ccdc44cc24fc875e19bc4f8640a72e3b89d1"} Nov 02 17:33:59.584274 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:59.584225 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-sjclp" podStartSLOduration=2.123176634 podStartE2EDuration="5.584209137s" totalImagesPullingTime="3.461032503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:59.581579097 +0000 UTC m=+258.202938886" watchObservedRunningTime="2026-11-02 17:33:59.584209137 +0000 UTC m=+258.205568915" Nov 02 17:33:59.865239 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:59.865205 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7"] Nov 02 17:33:59.875072 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:59.875042 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:33:59.897242 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:59.897209 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7"] Nov 02 17:33:59.898224 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:59.898170 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-pqqkt\"" Nov 02 17:33:59.946526 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:59.946490 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:33:59.946683 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:59.946541 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:33:59.946683 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:59.946624 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwt6g\" (UniqueName: \"kubernetes.io/projected/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-kube-api-access-rwt6g\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:33:59.946768 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:59.946717 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:33:59.946768 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:59.946746 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:33:59.946883 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:59.946783 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:33:59.946964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:59.946939 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:33:59.947091 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:59.946970 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:33:59.947091 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:33:59.947005 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.048412 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.048374 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.048581 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.048426 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.048581 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.048461 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.048581 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.048488 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.048581 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.048520 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.048789 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.048592 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwt6g\" (UniqueName: \"kubernetes.io/projected/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-kube-api-access-rwt6g\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.048789 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.048652 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.048789 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.048676 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.048789 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.048714 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.049020 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.048863 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.049020 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.048898 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.049186 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.049163 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.049353 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.049333 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.049489 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.049467 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.051443 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.051399 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.057357 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.057333 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.078289 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.078265 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwt6g\" (UniqueName: \"kubernetes.io/projected/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-kube-api-access-rwt6g\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.081687 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.081666 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac56f1a6-ffd6-4421-b2b9-8af5694db8b9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-gvxk7\" (UID: \"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.188655 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.188354 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:00.520596 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.520567 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7"] Nov 02 17:34:00.522201 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:34:00.522170 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac56f1a6_ffd6_4421_b2b9_8af5694db8b9.slice/crio-41b3ead9b45d0795b5dadbabf118cdb31e96fbb866c0117cf01a54f5678ae070 WatchSource:0}: Error finding container 41b3ead9b45d0795b5dadbabf118cdb31e96fbb866c0117cf01a54f5678ae070: Status 404 returned error can't find the container with id 41b3ead9b45d0795b5dadbabf118cdb31e96fbb866c0117cf01a54f5678ae070 Nov 02 17:34:00.524581 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.524547 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:34:00.524675 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.524616 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:34:00.524675 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.524658 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:34:00.539554 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.539525 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d6j7v" event={"ID":"d82bf74d-62d0-415d-aa93-abce6ae44ba7","Type":"ContainerStarted","Data":"aecaf6e96828106b4f118b0ae44ddf8d570f1b688935e7ead4e5a329543a6ecb"} Nov 02 17:34:00.539912 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.539606 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d6j7v" Nov 02 17:34:00.541356 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.541227 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" event={"ID":"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9","Type":"ContainerStarted","Data":"41b3ead9b45d0795b5dadbabf118cdb31e96fbb866c0117cf01a54f5678ae070"} Nov 02 17:34:00.588990 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.588950 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d6j7v" podStartSLOduration=0.969087741 podStartE2EDuration="2.588936167s" totalImagesPullingTime="1.619848426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:00.586677889 +0000 UTC m=+259.208037667" watchObservedRunningTime="2026-11-02 17:34:00.588936167 +0000 UTC m=+259.210295943" Nov 02 17:34:00.673195 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.673154 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bvtck"] Nov 02 17:34:00.675649 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.675627 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bvtck" Nov 02 17:34:00.723277 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.723239 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 17:34:00.723277 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.723271 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 17:34:00.723925 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.723904 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-rblgv\"" Nov 02 17:34:00.742937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.742905 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bvtck"] Nov 02 17:34:00.755223 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.755186 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btn56\" (UniqueName: \"kubernetes.io/projected/34275f86-46f6-4b1d-bd0e-865383413fcf-kube-api-access-btn56\") pod \"cert-manager-webhook-5dbbbf75cc-bvtck\" (UID: \"34275f86-46f6-4b1d-bd0e-865383413fcf\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bvtck" Nov 02 17:34:00.755398 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.755245 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34275f86-46f6-4b1d-bd0e-865383413fcf-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bvtck\" (UID: \"34275f86-46f6-4b1d-bd0e-865383413fcf\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bvtck" Nov 02 17:34:00.856031 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.855997 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btn56\" (UniqueName: \"kubernetes.io/projected/34275f86-46f6-4b1d-bd0e-865383413fcf-kube-api-access-btn56\") pod \"cert-manager-webhook-5dbbbf75cc-bvtck\" (UID: \"34275f86-46f6-4b1d-bd0e-865383413fcf\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bvtck" Nov 02 17:34:00.856191 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.856056 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34275f86-46f6-4b1d-bd0e-865383413fcf-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bvtck\" (UID: \"34275f86-46f6-4b1d-bd0e-865383413fcf\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bvtck" Nov 02 17:34:00.882117 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.882088 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34275f86-46f6-4b1d-bd0e-865383413fcf-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bvtck\" (UID: \"34275f86-46f6-4b1d-bd0e-865383413fcf\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bvtck" Nov 02 17:34:00.882358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.882335 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btn56\" (UniqueName: \"kubernetes.io/projected/34275f86-46f6-4b1d-bd0e-865383413fcf-kube-api-access-btn56\") pod \"cert-manager-webhook-5dbbbf75cc-bvtck\" (UID: \"34275f86-46f6-4b1d-bd0e-865383413fcf\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bvtck" Nov 02 17:34:00.988316 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:00.988280 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bvtck" Nov 02 17:34:01.310884 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:01.310837 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bvtck"] Nov 02 17:34:01.312245 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:34:01.312219 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34275f86_46f6_4b1d_bd0e_865383413fcf.slice/crio-26807890ebf2209c01c3887b9b15ee9bf03be41d77fb388c15f6909b76a758b2 WatchSource:0}: Error finding container 26807890ebf2209c01c3887b9b15ee9bf03be41d77fb388c15f6909b76a758b2: Status 404 returned error can't find the container with id 26807890ebf2209c01c3887b9b15ee9bf03be41d77fb388c15f6909b76a758b2 Nov 02 17:34:01.547983 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:01.547948 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" event={"ID":"ac56f1a6-ffd6-4421-b2b9-8af5694db8b9","Type":"ContainerStarted","Data":"e20dc78d6cf757a12b738422c6a8e8592225f39caa159149ebb75083ac430ac2"} Nov 02 17:34:01.549268 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:01.549242 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bvtck" event={"ID":"34275f86-46f6-4b1d-bd0e-865383413fcf","Type":"ContainerStarted","Data":"26807890ebf2209c01c3887b9b15ee9bf03be41d77fb388c15f6909b76a758b2"} Nov 02 17:34:01.598225 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:01.598132 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" podStartSLOduration=2.598118569 podStartE2EDuration="2.598118569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:01.596478338 +0000 UTC m=+260.217838115" watchObservedRunningTime="2026-11-02 17:34:01.598118569 +0000 UTC m=+260.219478345" Nov 02 17:34:02.189884 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:02.189836 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:02.195769 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:02.195745 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:02.429646 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:02.429609 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5zwtt"] Nov 02 17:34:02.432511 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:02.432488 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5zwtt" Nov 02 17:34:02.452563 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:02.452532 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5zwtt"] Nov 02 17:34:02.453111 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:02.453086 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kqm5s\"" Nov 02 17:34:02.469704 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:02.469666 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4nt8\" (UniqueName: \"kubernetes.io/projected/b792f0f9-8122-4134-a78b-b322dee30fd8-kube-api-access-q4nt8\") pod \"authorino-56c6c9ddd5-5zwtt\" (UID: \"b792f0f9-8122-4134-a78b-b322dee30fd8\") " pod="kuadrant-system/authorino-56c6c9ddd5-5zwtt" Nov 02 17:34:02.555739 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:02.555710 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:02.557064 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:02.557040 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-gvxk7" Nov 02 17:34:02.570551 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:02.570521 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4nt8\" (UniqueName: \"kubernetes.io/projected/b792f0f9-8122-4134-a78b-b322dee30fd8-kube-api-access-q4nt8\") pod \"authorino-56c6c9ddd5-5zwtt\" (UID: \"b792f0f9-8122-4134-a78b-b322dee30fd8\") " pod="kuadrant-system/authorino-56c6c9ddd5-5zwtt" Nov 02 17:34:02.601901 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:02.601872 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4nt8\" (UniqueName: \"kubernetes.io/projected/b792f0f9-8122-4134-a78b-b322dee30fd8-kube-api-access-q4nt8\") pod \"authorino-56c6c9ddd5-5zwtt\" (UID: \"b792f0f9-8122-4134-a78b-b322dee30fd8\") " pod="kuadrant-system/authorino-56c6c9ddd5-5zwtt" Nov 02 17:34:02.746138 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:02.746060 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5zwtt" Nov 02 17:34:03.066268 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:03.066235 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5zwtt"] Nov 02 17:34:03.067937 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:34:03.067889 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb792f0f9_8122_4134_a78b_b322dee30fd8.slice/crio-cbc1a2f14cee7c23288d8b2238bfc360ecaf5d25abe45770af04802aed2782a1 WatchSource:0}: Error finding container cbc1a2f14cee7c23288d8b2238bfc360ecaf5d25abe45770af04802aed2782a1: Status 404 returned error can't find the container with id cbc1a2f14cee7c23288d8b2238bfc360ecaf5d25abe45770af04802aed2782a1 Nov 02 17:34:03.563068 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:03.563001 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5zwtt" event={"ID":"b792f0f9-8122-4134-a78b-b322dee30fd8","Type":"ContainerStarted","Data":"cbc1a2f14cee7c23288d8b2238bfc360ecaf5d25abe45770af04802aed2782a1"} Nov 02 17:34:04.569502 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:04.569461 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bvtck" event={"ID":"34275f86-46f6-4b1d-bd0e-865383413fcf","Type":"ContainerStarted","Data":"e93c9c4670db9a2dc9e52311ce16e0d4ca2d50d4a0185d0319835a654500c88d"} Nov 02 17:34:04.569977 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:04.569736 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bvtck" Nov 02 17:34:04.611916 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:04.611842 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bvtck" podStartSLOduration=1.997655443 podStartE2EDuration="4.611828794s" totalImagesPullingTime="2.614173351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:04.610443244 +0000 UTC m=+263.231803017" watchObservedRunningTime="2026-11-02 17:34:04.611828794 +0000 UTC m=+263.233188571" Nov 02 17:34:05.509859 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:05.509828 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj" Nov 02 17:34:07.588075 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:07.588031 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5zwtt" event={"ID":"b792f0f9-8122-4134-a78b-b322dee30fd8","Type":"ContainerStarted","Data":"3c71c6e07c66ad13e85e5dd875b987fea16b642eb37ec6546aee7cc2d515b083"} Nov 02 17:34:07.626176 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:07.626123 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-5zwtt" podStartSLOduration=1.760180198 podStartE2EDuration="5.62610902s" totalImagesPullingTime="3.865928822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:07.625960242 +0000 UTC m=+266.247320018" watchObservedRunningTime="2026-11-02 17:34:07.62610902 +0000 UTC m=+266.247468796" Nov 02 17:34:10.543830 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:10.543793 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-sjclp" Nov 02 17:34:10.577337 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:10.577307 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bvtck" Nov 02 17:34:11.552166 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:11.552140 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d6j7v" Nov 02 17:34:21.769724 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:34:21.769683 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:34:21.937705 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:21.937668 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj"] Nov 02 17:34:21.938001 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:21.937972 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj" podUID="a049234f-a867-41f5-969f-5ac24f6bfb13" containerName="manager" containerID="cri-o://7b2de1a3482036a5b787d26ecdb6eca907ea8f514f8a477512569fd61fbfb30e" gracePeriod=10 Nov 02 17:34:22.178516 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.178494 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj" Nov 02 17:34:22.252305 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.252272 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a049234f-a867-41f5-969f-5ac24f6bfb13-kube-api-access-nxtb8\" (UniqueName: \"kubernetes.io/projected/a049234f-a867-41f5-969f-5ac24f6bfb13-kube-api-access-nxtb8\") pod \"a049234f-a867-41f5-969f-5ac24f6bfb13\" (UID: \"a049234f-a867-41f5-969f-5ac24f6bfb13\") " Nov 02 17:34:22.252481 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.252328 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a049234f-a867-41f5-969f-5ac24f6bfb13-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a049234f-a867-41f5-969f-5ac24f6bfb13-extensions-socket-volume\") pod \"a049234f-a867-41f5-969f-5ac24f6bfb13\" (UID: \"a049234f-a867-41f5-969f-5ac24f6bfb13\") " Nov 02 17:34:22.252876 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.252822 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a049234f-a867-41f5-969f-5ac24f6bfb13-extensions-socket-volume" pod "a049234f-a867-41f5-969f-5ac24f6bfb13" (UID: "a049234f-a867-41f5-969f-5ac24f6bfb13"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:34:22.254450 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.254420 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a049234f-a867-41f5-969f-5ac24f6bfb13-kube-api-access-nxtb8" pod "a049234f-a867-41f5-969f-5ac24f6bfb13" (UID: "a049234f-a867-41f5-969f-5ac24f6bfb13"). InnerVolumeSpecName "kube-api-access-nxtb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:34:22.353277 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.353206 4860 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a049234f-a867-41f5-969f-5ac24f6bfb13-extensions-socket-volume\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:34:22.353277 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.353230 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxtb8\" (UniqueName: \"kubernetes.io/projected/a049234f-a867-41f5-969f-5ac24f6bfb13-kube-api-access-nxtb8\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:34:22.367280 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.367248 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-8zvgb"] Nov 02 17:34:22.367611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.367599 4860 cpu_manager.go:414] "containerMap: removing container" podUID="a049234f-a867-41f5-969f-5ac24f6bfb13" containerName="manager" Nov 02 17:34:22.367652 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.367613 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a049234f-a867-41f5-969f-5ac24f6bfb13" containerName="manager" Nov 02 17:34:22.367705 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.367696 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a049234f-a867-41f5-969f-5ac24f6bfb13" containerName="manager" Nov 02 17:34:22.369921 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.369902 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8zvgb" Nov 02 17:34:22.390014 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.389985 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-8zvgb"] Nov 02 17:34:22.393935 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.393907 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 17:34:22.394063 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.393998 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 17:34:22.394176 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.394153 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-42hvb\"" Nov 02 17:34:22.394332 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.394252 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 17:34:22.454651 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.454618 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54j7m\" (UniqueName: \"kubernetes.io/projected/11899ead-b87a-4f5e-9b8a-f2325cf41709-kube-api-access-54j7m\") pod \"postgres-7cdbf6dc44-8zvgb\" (UID: \"11899ead-b87a-4f5e-9b8a-f2325cf41709\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8zvgb" Nov 02 17:34:22.454651 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.454663 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/11899ead-b87a-4f5e-9b8a-f2325cf41709-data\") pod \"postgres-7cdbf6dc44-8zvgb\" (UID: \"11899ead-b87a-4f5e-9b8a-f2325cf41709\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8zvgb" Nov 02 17:34:22.555550 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.555502 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54j7m\" (UniqueName: \"kubernetes.io/projected/11899ead-b87a-4f5e-9b8a-f2325cf41709-kube-api-access-54j7m\") pod \"postgres-7cdbf6dc44-8zvgb\" (UID: \"11899ead-b87a-4f5e-9b8a-f2325cf41709\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8zvgb" Nov 02 17:34:22.555746 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.555563 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/11899ead-b87a-4f5e-9b8a-f2325cf41709-data\") pod \"postgres-7cdbf6dc44-8zvgb\" (UID: \"11899ead-b87a-4f5e-9b8a-f2325cf41709\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8zvgb" Nov 02 17:34:22.556063 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.556039 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/11899ead-b87a-4f5e-9b8a-f2325cf41709-data\") pod \"postgres-7cdbf6dc44-8zvgb\" (UID: \"11899ead-b87a-4f5e-9b8a-f2325cf41709\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8zvgb" Nov 02 17:34:22.585008 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.584978 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54j7m\" (UniqueName: \"kubernetes.io/projected/11899ead-b87a-4f5e-9b8a-f2325cf41709-kube-api-access-54j7m\") pod \"postgres-7cdbf6dc44-8zvgb\" (UID: \"11899ead-b87a-4f5e-9b8a-f2325cf41709\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8zvgb" Nov 02 17:34:22.679964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.679883 4860 generic.go:356] "Generic (PLEG): container finished" podID="a049234f-a867-41f5-969f-5ac24f6bfb13" containerID="7b2de1a3482036a5b787d26ecdb6eca907ea8f514f8a477512569fd61fbfb30e" exitCode=0 Nov 02 17:34:22.679964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.679937 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj" Nov 02 17:34:22.680129 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.679976 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj" event={"ID":"a049234f-a867-41f5-969f-5ac24f6bfb13","Type":"ContainerDied","Data":"7b2de1a3482036a5b787d26ecdb6eca907ea8f514f8a477512569fd61fbfb30e"} Nov 02 17:34:22.680129 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.680027 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj" event={"ID":"a049234f-a867-41f5-969f-5ac24f6bfb13","Type":"ContainerDied","Data":"55f9702619aedea92682145c50f33e80be2e7b6d8dc0d2d8b6aa7333518119ed"} Nov 02 17:34:22.680129 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.680052 4860 scope.go:122] "RemoveContainer" containerID="7b2de1a3482036a5b787d26ecdb6eca907ea8f514f8a477512569fd61fbfb30e" Nov 02 17:34:22.682253 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.682223 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8zvgb" Nov 02 17:34:22.689952 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.689931 4860 scope.go:122] "RemoveContainer" containerID="7b2de1a3482036a5b787d26ecdb6eca907ea8f514f8a477512569fd61fbfb30e" Nov 02 17:34:22.690238 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:34:22.690210 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b2de1a3482036a5b787d26ecdb6eca907ea8f514f8a477512569fd61fbfb30e\": container with ID starting with 7b2de1a3482036a5b787d26ecdb6eca907ea8f514f8a477512569fd61fbfb30e not found: ID does not exist" containerID="7b2de1a3482036a5b787d26ecdb6eca907ea8f514f8a477512569fd61fbfb30e" Nov 02 17:34:22.690308 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.690246 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b2de1a3482036a5b787d26ecdb6eca907ea8f514f8a477512569fd61fbfb30e"} err="failed to get container status \"7b2de1a3482036a5b787d26ecdb6eca907ea8f514f8a477512569fd61fbfb30e\": rpc error: code = NotFound desc = could not find container \"7b2de1a3482036a5b787d26ecdb6eca907ea8f514f8a477512569fd61fbfb30e\": container with ID starting with 7b2de1a3482036a5b787d26ecdb6eca907ea8f514f8a477512569fd61fbfb30e not found: ID does not exist" Nov 02 17:34:22.754228 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.754178 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj"] Nov 02 17:34:22.760486 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.760453 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rs7mj"] Nov 02 17:34:22.997676 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:22.997643 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-8zvgb"] Nov 02 17:34:22.999185 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:34:22.999156 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11899ead_b87a_4f5e_9b8a_f2325cf41709.slice/crio-088be3f58f35370e3f08d0c37686506eead0ce7b5977d9da54ea16b8bc4fbbed WatchSource:0}: Error finding container 088be3f58f35370e3f08d0c37686506eead0ce7b5977d9da54ea16b8bc4fbbed: Status 404 returned error can't find the container with id 088be3f58f35370e3f08d0c37686506eead0ce7b5977d9da54ea16b8bc4fbbed Nov 02 17:34:23.497829 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:23.497798 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a049234f-a867-41f5-969f-5ac24f6bfb13" path="/var/lib/kubelet/pods/a049234f-a867-41f5-969f-5ac24f6bfb13/volumes" Nov 02 17:34:23.686925 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:23.686887 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8zvgb" event={"ID":"11899ead-b87a-4f5e-9b8a-f2325cf41709","Type":"ContainerStarted","Data":"088be3f58f35370e3f08d0c37686506eead0ce7b5977d9da54ea16b8bc4fbbed"} Nov 02 17:34:28.721426 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:28.721385 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8zvgb" event={"ID":"11899ead-b87a-4f5e-9b8a-f2325cf41709","Type":"ContainerStarted","Data":"1eaca8b5dd6751e4331f014853e8609511f04632d43d4b57d8ddb7c292e92b90"} Nov 02 17:34:28.721784 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:28.721470 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8zvgb" Nov 02 17:34:28.766373 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:28.766319 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8zvgb" podStartSLOduration=1.215337168 podStartE2EDuration="6.766304494s" totalImagesPullingTime="5.550967326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:28.764517662 +0000 UTC m=+287.385877444" watchObservedRunningTime="2026-11-02 17:34:28.766304494 +0000 UTC m=+287.387664269" Nov 02 17:34:34.756532 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:34.756506 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8zvgb" Nov 02 17:34:41.508462 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:41.508435 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 17:34:41.508890 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:41.508439 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 17:34:41.526374 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:41.526350 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 17:34:41.526493 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:41.526435 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 17:34:42.402713 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.402680 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-fljqr"] Nov 02 17:34:42.408946 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.408922 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-fljqr" Nov 02 17:34:42.433003 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.432968 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc81af84-6487-4374-833b-b7372e323a87-metrics-certs\") pod \"maas-controller-579cf7c6d-fljqr\" (UID: \"fc81af84-6487-4374-833b-b7372e323a87\") " pod="opendatahub/maas-controller-579cf7c6d-fljqr" Nov 02 17:34:42.433152 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.433054 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cths6\" (UniqueName: \"kubernetes.io/projected/fc81af84-6487-4374-833b-b7372e323a87-kube-api-access-cths6\") pod \"maas-controller-579cf7c6d-fljqr\" (UID: \"fc81af84-6487-4374-833b-b7372e323a87\") " pod="opendatahub/maas-controller-579cf7c6d-fljqr" Nov 02 17:34:42.433152 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.433098 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc81af84-6487-4374-833b-b7372e323a87-webhook-cert\") pod \"maas-controller-579cf7c6d-fljqr\" (UID: \"fc81af84-6487-4374-833b-b7372e323a87\") " pod="opendatahub/maas-controller-579cf7c6d-fljqr" Nov 02 17:34:42.437715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.437687 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 17:34:42.437715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.437705 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 17:34:42.437935 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.437767 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 17:34:42.437935 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.437824 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 17:34:42.437935 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.437833 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 17:34:42.437935 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.437836 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kv77t\"" Nov 02 17:34:42.440440 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.440419 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-fljqr"] Nov 02 17:34:42.533774 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.533732 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cths6\" (UniqueName: \"kubernetes.io/projected/fc81af84-6487-4374-833b-b7372e323a87-kube-api-access-cths6\") pod \"maas-controller-579cf7c6d-fljqr\" (UID: \"fc81af84-6487-4374-833b-b7372e323a87\") " pod="opendatahub/maas-controller-579cf7c6d-fljqr" Nov 02 17:34:42.534376 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.533782 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc81af84-6487-4374-833b-b7372e323a87-webhook-cert\") pod \"maas-controller-579cf7c6d-fljqr\" (UID: \"fc81af84-6487-4374-833b-b7372e323a87\") " pod="opendatahub/maas-controller-579cf7c6d-fljqr" Nov 02 17:34:42.534376 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.533947 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc81af84-6487-4374-833b-b7372e323a87-metrics-certs\") pod \"maas-controller-579cf7c6d-fljqr\" (UID: \"fc81af84-6487-4374-833b-b7372e323a87\") " pod="opendatahub/maas-controller-579cf7c6d-fljqr" Nov 02 17:34:42.534376 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:34:42.534130 4860 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Nov 02 17:34:42.534376 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:34:42.534208 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc81af84-6487-4374-833b-b7372e323a87-metrics-certs podName:fc81af84-6487-4374-833b-b7372e323a87 nodeName:}" failed. No retries permitted until 2026-11-02 17:34:43.034185023 +0000 UTC m=+301.655544792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc81af84-6487-4374-833b-b7372e323a87-metrics-certs") pod "maas-controller-579cf7c6d-fljqr" (UID: "fc81af84-6487-4374-833b-b7372e323a87") : secret "maas-controller-metrics-tls" not found Nov 02 17:34:42.536469 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.536444 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc81af84-6487-4374-833b-b7372e323a87-webhook-cert\") pod \"maas-controller-579cf7c6d-fljqr\" (UID: \"fc81af84-6487-4374-833b-b7372e323a87\") " pod="opendatahub/maas-controller-579cf7c6d-fljqr" Nov 02 17:34:42.559543 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.559512 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cths6\" (UniqueName: \"kubernetes.io/projected/fc81af84-6487-4374-833b-b7372e323a87-kube-api-access-cths6\") pod \"maas-controller-579cf7c6d-fljqr\" (UID: \"fc81af84-6487-4374-833b-b7372e323a87\") " pod="opendatahub/maas-controller-579cf7c6d-fljqr" Nov 02 17:34:42.772365 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.772287 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67b9f44df6-zzhv5"] Nov 02 17:34:42.781626 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.781586 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b9f44df6-zzhv5" Nov 02 17:34:42.797353 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.797330 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67b9f44df6-zzhv5"] Nov 02 17:34:42.836116 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.836087 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-metrics-certs\") pod \"maas-controller-67b9f44df6-zzhv5\" (UID: \"e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca\") " pod="opendatahub/maas-controller-67b9f44df6-zzhv5" Nov 02 17:34:42.836248 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.836152 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-webhook-cert\") pod \"maas-controller-67b9f44df6-zzhv5\" (UID: \"e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca\") " pod="opendatahub/maas-controller-67b9f44df6-zzhv5" Nov 02 17:34:42.836295 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.836282 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t229\" (UniqueName: \"kubernetes.io/projected/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-kube-api-access-9t229\") pod \"maas-controller-67b9f44df6-zzhv5\" (UID: \"e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca\") " pod="opendatahub/maas-controller-67b9f44df6-zzhv5" Nov 02 17:34:42.937425 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.937394 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t229\" (UniqueName: \"kubernetes.io/projected/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-kube-api-access-9t229\") pod \"maas-controller-67b9f44df6-zzhv5\" (UID: \"e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca\") " pod="opendatahub/maas-controller-67b9f44df6-zzhv5" Nov 02 17:34:42.937587 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.937432 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-metrics-certs\") pod \"maas-controller-67b9f44df6-zzhv5\" (UID: \"e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca\") " pod="opendatahub/maas-controller-67b9f44df6-zzhv5" Nov 02 17:34:42.937587 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.937490 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-webhook-cert\") pod \"maas-controller-67b9f44df6-zzhv5\" (UID: \"e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca\") " pod="opendatahub/maas-controller-67b9f44df6-zzhv5" Nov 02 17:34:42.939983 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.939958 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-metrics-certs\") pod \"maas-controller-67b9f44df6-zzhv5\" (UID: \"e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca\") " pod="opendatahub/maas-controller-67b9f44df6-zzhv5" Nov 02 17:34:42.939983 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.939972 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-webhook-cert\") pod \"maas-controller-67b9f44df6-zzhv5\" (UID: \"e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca\") " pod="opendatahub/maas-controller-67b9f44df6-zzhv5" Nov 02 17:34:42.960470 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:42.960446 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t229\" (UniqueName: \"kubernetes.io/projected/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-kube-api-access-9t229\") pod \"maas-controller-67b9f44df6-zzhv5\" (UID: \"e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca\") " pod="opendatahub/maas-controller-67b9f44df6-zzhv5" Nov 02 17:34:43.038724 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:43.038645 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc81af84-6487-4374-833b-b7372e323a87-metrics-certs\") pod \"maas-controller-579cf7c6d-fljqr\" (UID: \"fc81af84-6487-4374-833b-b7372e323a87\") " pod="opendatahub/maas-controller-579cf7c6d-fljqr" Nov 02 17:34:43.040958 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:43.040935 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc81af84-6487-4374-833b-b7372e323a87-metrics-certs\") pod \"maas-controller-579cf7c6d-fljqr\" (UID: \"fc81af84-6487-4374-833b-b7372e323a87\") " pod="opendatahub/maas-controller-579cf7c6d-fljqr" Nov 02 17:34:43.095103 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:43.095067 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b9f44df6-zzhv5" Nov 02 17:34:43.321749 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:43.321715 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-fljqr" Nov 02 17:34:43.393998 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:43.393974 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67b9f44df6-zzhv5"] Nov 02 17:34:43.395508 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:34:43.395482 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55d5ea9_6fb8_4f82_a9a9_076d69a0f0ca.slice/crio-075be13926602f638ef5a966b9b995278e87c3ae4954386426835d1b186367f9 WatchSource:0}: Error finding container 075be13926602f638ef5a966b9b995278e87c3ae4954386426835d1b186367f9: Status 404 returned error can't find the container with id 075be13926602f638ef5a966b9b995278e87c3ae4954386426835d1b186367f9 Nov 02 17:34:43.620444 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:43.620416 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-fljqr"] Nov 02 17:34:43.621808 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:34:43.621780 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc81af84_6487_4374_833b_b7372e323a87.slice/crio-2f664eb609fd7168f85e2be77b7a8703cba7fd87e0352013f751b237abf23380 WatchSource:0}: Error finding container 2f664eb609fd7168f85e2be77b7a8703cba7fd87e0352013f751b237abf23380: Status 404 returned error can't find the container with id 2f664eb609fd7168f85e2be77b7a8703cba7fd87e0352013f751b237abf23380 Nov 02 17:34:43.799783 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:43.799747 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-fljqr" event={"ID":"fc81af84-6487-4374-833b-b7372e323a87","Type":"ContainerStarted","Data":"2f664eb609fd7168f85e2be77b7a8703cba7fd87e0352013f751b237abf23380"} Nov 02 17:34:43.801032 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:43.800993 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b9f44df6-zzhv5" event={"ID":"e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca","Type":"ContainerStarted","Data":"075be13926602f638ef5a966b9b995278e87c3ae4954386426835d1b186367f9"} Nov 02 17:34:46.825057 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:46.825013 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-fljqr" event={"ID":"fc81af84-6487-4374-833b-b7372e323a87","Type":"ContainerStarted","Data":"a62cdca1617fb2487126ee11bb1ff6a080d96a90172fb6f1a0f86524b9712f9a"} Nov 02 17:34:46.825484 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:46.825085 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-fljqr" Nov 02 17:34:46.826820 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:46.826793 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b9f44df6-zzhv5" event={"ID":"e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca","Type":"ContainerStarted","Data":"b260e74f62f4b3be15b494726b015b2a7fce41d3d3f9acdf60d95ba321a53018"} Nov 02 17:34:46.826942 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:46.826908 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67b9f44df6-zzhv5" Nov 02 17:34:46.868048 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:46.867993 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-fljqr" podStartSLOduration=2.11986809 podStartE2EDuration="4.867980549s" totalImagesPullingTime="2.748112459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:46.867415655 +0000 UTC m=+305.488775431" watchObservedRunningTime="2026-11-02 17:34:46.867980549 +0000 UTC m=+305.489340324" Nov 02 17:34:46.912607 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:46.912554 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-67b9f44df6-zzhv5" podStartSLOduration=2.127183514 podStartE2EDuration="4.912539882s" totalImagesPullingTime="2.785356368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:46.911157811 +0000 UTC m=+305.532517588" watchObservedRunningTime="2026-11-02 17:34:46.912539882 +0000 UTC m=+305.533899657" Nov 02 17:34:47.942479 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:47.942441 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm"] Nov 02 17:34:47.946066 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:47.946048 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" Nov 02 17:34:47.973024 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:47.972996 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm"] Nov 02 17:34:47.982136 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:47.982054 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 02 17:34:47.982294 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:47.982149 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 02 17:34:47.982294 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:47.982190 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-hsv8w\"" Nov 02 17:34:47.982294 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:47.982241 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 02 17:34:47.983493 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:47.983467 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/cd9cd220-880d-4fd7-91ba-cc7bb242867e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bm8dm\" (UID: \"cd9cd220-880d-4fd7-91ba-cc7bb242867e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" Nov 02 17:34:47.983635 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:47.983535 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cd9cd220-880d-4fd7-91ba-cc7bb242867e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bm8dm\" (UID: \"cd9cd220-880d-4fd7-91ba-cc7bb242867e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" Nov 02 17:34:47.983635 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:47.983618 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9cd220-880d-4fd7-91ba-cc7bb242867e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bm8dm\" (UID: \"cd9cd220-880d-4fd7-91ba-cc7bb242867e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" Nov 02 17:34:47.983720 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:47.983652 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29kxq\" (UniqueName: \"kubernetes.io/projected/cd9cd220-880d-4fd7-91ba-cc7bb242867e-kube-api-access-29kxq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bm8dm\" (UID: \"cd9cd220-880d-4fd7-91ba-cc7bb242867e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" Nov 02 17:34:48.084721 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:48.084691 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9cd220-880d-4fd7-91ba-cc7bb242867e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bm8dm\" (UID: \"cd9cd220-880d-4fd7-91ba-cc7bb242867e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" Nov 02 17:34:48.084939 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:48.084736 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29kxq\" (UniqueName: \"kubernetes.io/projected/cd9cd220-880d-4fd7-91ba-cc7bb242867e-kube-api-access-29kxq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bm8dm\" (UID: \"cd9cd220-880d-4fd7-91ba-cc7bb242867e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" Nov 02 17:34:48.084939 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:48.084874 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/cd9cd220-880d-4fd7-91ba-cc7bb242867e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bm8dm\" (UID: \"cd9cd220-880d-4fd7-91ba-cc7bb242867e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" Nov 02 17:34:48.085074 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:34:48.084971 4860 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 02 17:34:48.085074 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:48.084980 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cd9cd220-880d-4fd7-91ba-cc7bb242867e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bm8dm\" (UID: \"cd9cd220-880d-4fd7-91ba-cc7bb242867e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" Nov 02 17:34:48.085074 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:34:48.085024 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd9cd220-880d-4fd7-91ba-cc7bb242867e-proxy-source podName:cd9cd220-880d-4fd7-91ba-cc7bb242867e nodeName:}" failed. No retries permitted until 2026-11-02 17:34:48.585008812 +0000 UTC m=+307.206368605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/cd9cd220-880d-4fd7-91ba-cc7bb242867e-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" (UID: "cd9cd220-880d-4fd7-91ba-cc7bb242867e") : configmap "usage-tenancy-proxy-source" not found Nov 02 17:34:48.085243 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:34:48.085082 4860 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 02 17:34:48.085243 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:34:48.085134 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd9cd220-880d-4fd7-91ba-cc7bb242867e-tls-cert podName:cd9cd220-880d-4fd7-91ba-cc7bb242867e nodeName:}" failed. No retries permitted until 2026-11-02 17:34:48.585118841 +0000 UTC m=+307.206478609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/cd9cd220-880d-4fd7-91ba-cc7bb242867e-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" (UID: "cd9cd220-880d-4fd7-91ba-cc7bb242867e") : secret "usage-logs-tenancy-proxy-tls" not found Nov 02 17:34:48.085488 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:48.085465 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9cd220-880d-4fd7-91ba-cc7bb242867e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bm8dm\" (UID: \"cd9cd220-880d-4fd7-91ba-cc7bb242867e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" Nov 02 17:34:48.108486 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:48.108452 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29kxq\" (UniqueName: \"kubernetes.io/projected/cd9cd220-880d-4fd7-91ba-cc7bb242867e-kube-api-access-29kxq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bm8dm\" (UID: \"cd9cd220-880d-4fd7-91ba-cc7bb242867e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" Nov 02 17:34:48.588798 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:48.588755 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cd9cd220-880d-4fd7-91ba-cc7bb242867e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bm8dm\" (UID: \"cd9cd220-880d-4fd7-91ba-cc7bb242867e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" Nov 02 17:34:48.589043 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:48.589014 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/cd9cd220-880d-4fd7-91ba-cc7bb242867e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bm8dm\" (UID: \"cd9cd220-880d-4fd7-91ba-cc7bb242867e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" Nov 02 17:34:48.589741 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:48.589713 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/cd9cd220-880d-4fd7-91ba-cc7bb242867e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bm8dm\" (UID: \"cd9cd220-880d-4fd7-91ba-cc7bb242867e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" Nov 02 17:34:48.591357 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:48.591336 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cd9cd220-880d-4fd7-91ba-cc7bb242867e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bm8dm\" (UID: \"cd9cd220-880d-4fd7-91ba-cc7bb242867e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" Nov 02 17:34:48.859020 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:48.858941 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" Nov 02 17:34:49.158446 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:49.158403 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm"] Nov 02 17:34:49.159863 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:34:49.159811 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9cd220_880d_4fd7_91ba_cc7bb242867e.slice/crio-6bea6ac74c711c93710d19e408f78ec8ae7a854344c7b9ff2eaac1f3104425bf WatchSource:0}: Error finding container 6bea6ac74c711c93710d19e408f78ec8ae7a854344c7b9ff2eaac1f3104425bf: Status 404 returned error can't find the container with id 6bea6ac74c711c93710d19e408f78ec8ae7a854344c7b9ff2eaac1f3104425bf Nov 02 17:34:49.161834 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:49.161813 4860 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:34:49.848297 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:49.848255 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" event={"ID":"cd9cd220-880d-4fd7-91ba-cc7bb242867e","Type":"ContainerStarted","Data":"6bea6ac74c711c93710d19e408f78ec8ae7a854344c7b9ff2eaac1f3104425bf"} Nov 02 17:34:51.776545 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:34:51.776465 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:34:57.836358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:57.836124 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67b9f44df6-zzhv5" Nov 02 17:34:57.836358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:57.836235 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-fljqr" Nov 02 17:34:57.974384 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:57.974351 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-fljqr"] Nov 02 17:34:57.974610 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:57.974578 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-fljqr" podUID="fc81af84-6487-4374-833b-b7372e323a87" containerName="manager" containerID="cri-o://a62cdca1617fb2487126ee11bb1ff6a080d96a90172fb6f1a0f86524b9712f9a" gracePeriod=30 Nov 02 17:34:58.250179 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:58.250154 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-fljqr" Nov 02 17:34:58.388180 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:58.388103 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc81af84-6487-4374-833b-b7372e323a87-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc81af84-6487-4374-833b-b7372e323a87-metrics-certs\") pod \"fc81af84-6487-4374-833b-b7372e323a87\" (UID: \"fc81af84-6487-4374-833b-b7372e323a87\") " Nov 02 17:34:58.388180 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:58.388166 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc81af84-6487-4374-833b-b7372e323a87-kube-api-access-cths6\" (UniqueName: \"kubernetes.io/projected/fc81af84-6487-4374-833b-b7372e323a87-kube-api-access-cths6\") pod \"fc81af84-6487-4374-833b-b7372e323a87\" (UID: \"fc81af84-6487-4374-833b-b7372e323a87\") " Nov 02 17:34:58.388373 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:58.388189 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc81af84-6487-4374-833b-b7372e323a87-webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc81af84-6487-4374-833b-b7372e323a87-webhook-cert\") pod \"fc81af84-6487-4374-833b-b7372e323a87\" (UID: \"fc81af84-6487-4374-833b-b7372e323a87\") " Nov 02 17:34:58.390746 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:58.390688 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc81af84-6487-4374-833b-b7372e323a87-webhook-cert" pod "fc81af84-6487-4374-833b-b7372e323a87" (UID: "fc81af84-6487-4374-833b-b7372e323a87"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:58.390909 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:58.390821 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc81af84-6487-4374-833b-b7372e323a87-metrics-certs" pod "fc81af84-6487-4374-833b-b7372e323a87" (UID: "fc81af84-6487-4374-833b-b7372e323a87"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:58.390909 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:58.390887 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc81af84-6487-4374-833b-b7372e323a87-kube-api-access-cths6" pod "fc81af84-6487-4374-833b-b7372e323a87" (UID: "fc81af84-6487-4374-833b-b7372e323a87"). InnerVolumeSpecName "kube-api-access-cths6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:34:58.489488 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:58.489450 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc81af84-6487-4374-833b-b7372e323a87-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:34:58.489488 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:58.489483 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cths6\" (UniqueName: \"kubernetes.io/projected/fc81af84-6487-4374-833b-b7372e323a87-kube-api-access-cths6\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:34:58.489488 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:58.489493 4860 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc81af84-6487-4374-833b-b7372e323a87-webhook-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:34:58.916128 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:58.916094 4860 generic.go:356] "Generic (PLEG): container finished" podID="fc81af84-6487-4374-833b-b7372e323a87" containerID="a62cdca1617fb2487126ee11bb1ff6a080d96a90172fb6f1a0f86524b9712f9a" exitCode=0 Nov 02 17:34:58.916574 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:58.916165 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-fljqr" Nov 02 17:34:58.916574 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:58.916195 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-fljqr" event={"ID":"fc81af84-6487-4374-833b-b7372e323a87","Type":"ContainerDied","Data":"a62cdca1617fb2487126ee11bb1ff6a080d96a90172fb6f1a0f86524b9712f9a"} Nov 02 17:34:58.916574 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:58.916271 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-fljqr" event={"ID":"fc81af84-6487-4374-833b-b7372e323a87","Type":"ContainerDied","Data":"2f664eb609fd7168f85e2be77b7a8703cba7fd87e0352013f751b237abf23380"} Nov 02 17:34:58.916574 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:58.916299 4860 scope.go:122] "RemoveContainer" containerID="a62cdca1617fb2487126ee11bb1ff6a080d96a90172fb6f1a0f86524b9712f9a" Nov 02 17:34:58.929448 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:58.929429 4860 scope.go:122] "RemoveContainer" containerID="a62cdca1617fb2487126ee11bb1ff6a080d96a90172fb6f1a0f86524b9712f9a" Nov 02 17:34:58.929751 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:34:58.929728 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a62cdca1617fb2487126ee11bb1ff6a080d96a90172fb6f1a0f86524b9712f9a\": container with ID starting with a62cdca1617fb2487126ee11bb1ff6a080d96a90172fb6f1a0f86524b9712f9a not found: ID does not exist" containerID="a62cdca1617fb2487126ee11bb1ff6a080d96a90172fb6f1a0f86524b9712f9a" Nov 02 17:34:58.929808 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:58.929765 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a62cdca1617fb2487126ee11bb1ff6a080d96a90172fb6f1a0f86524b9712f9a"} err="failed to get container status \"a62cdca1617fb2487126ee11bb1ff6a080d96a90172fb6f1a0f86524b9712f9a\": rpc error: code = NotFound desc = could not find container \"a62cdca1617fb2487126ee11bb1ff6a080d96a90172fb6f1a0f86524b9712f9a\": container with ID starting with a62cdca1617fb2487126ee11bb1ff6a080d96a90172fb6f1a0f86524b9712f9a not found: ID does not exist" Nov 02 17:34:58.972778 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:58.972741 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-fljqr"] Nov 02 17:34:58.983587 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:58.983550 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-fljqr"] Nov 02 17:34:59.499638 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:34:59.499601 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc81af84-6487-4374-833b-b7372e323a87" path="/var/lib/kubelet/pods/fc81af84-6487-4374-833b-b7372e323a87/volumes" Nov 02 17:35:00.802895 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.802839 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-gt7zt"] Nov 02 17:35:00.803547 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.803424 4860 cpu_manager.go:414] "containerMap: removing container" podUID="fc81af84-6487-4374-833b-b7372e323a87" containerName="manager" Nov 02 17:35:00.803547 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.803446 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc81af84-6487-4374-833b-b7372e323a87" containerName="manager" Nov 02 17:35:00.803692 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.803572 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc81af84-6487-4374-833b-b7372e323a87" containerName="manager" Nov 02 17:35:00.806942 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.806916 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-gt7zt" Nov 02 17:35:00.828764 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.828732 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-gt7zt"] Nov 02 17:35:00.863930 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.863907 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz"] Nov 02 17:35:00.869008 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.868988 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz" Nov 02 17:35:00.869438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.869419 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-pb8bb\"" Nov 02 17:35:00.869522 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.869495 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 02 17:35:00.890159 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.890127 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz"] Nov 02 17:35:00.900508 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.900481 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-h8bxb"] Nov 02 17:35:00.911887 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.911838 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-h8bxb" Nov 02 17:35:00.913018 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.912989 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxl8t\" (UniqueName: \"kubernetes.io/projected/412157d1-0980-4139-bff7-466a1df1b0d3-kube-api-access-lxl8t\") pod \"kserve-controller-manager-776cc66d47-gt7zt\" (UID: \"412157d1-0980-4139-bff7-466a1df1b0d3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-gt7zt" Nov 02 17:35:00.913165 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.913141 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/412157d1-0980-4139-bff7-466a1df1b0d3-cert\") pod \"kserve-controller-manager-776cc66d47-gt7zt\" (UID: \"412157d1-0980-4139-bff7-466a1df1b0d3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-gt7zt" Nov 02 17:35:00.930995 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.930960 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-h8bxb"] Nov 02 17:35:00.939230 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.939200 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" event={"ID":"cd9cd220-880d-4fd7-91ba-cc7bb242867e","Type":"ContainerStarted","Data":"d59c20790ad95ada4a7f5ab3461df7c29ad2db8995ba50b3e1801797f2866282"} Nov 02 17:35:00.939335 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.939251 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" Nov 02 17:35:00.954125 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.954096 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-27r8m"] Nov 02 17:35:00.957810 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.957786 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-27r8m" Nov 02 17:35:00.968682 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.968652 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 02 17:35:00.968796 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.968678 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 17:35:00.968796 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.968755 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-lfqqc\"" Nov 02 17:35:00.968945 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.968871 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-vxt5t\"" Nov 02 17:35:00.981266 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:00.981242 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-27r8m"] Nov 02 17:35:01.013748 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.013712 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tswzj\" (UniqueName: \"kubernetes.io/projected/f2fa7be0-9911-423c-adad-e88db47815a0-kube-api-access-tswzj\") pod \"odh-model-controller-769875db56-27r8m\" (UID: \"f2fa7be0-9911-423c-adad-e88db47815a0\") " pod="opendatahub/odh-model-controller-769875db56-27r8m" Nov 02 17:35:01.013916 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.013788 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf7lc\" (UniqueName: \"kubernetes.io/projected/9e543ec4-db22-4e57-bad4-ef8b2e1920e7-kube-api-access-zf7lc\") pod \"model-serving-api-75bd89fdfc-h8bxb\" (UID: \"9e543ec4-db22-4e57-bad4-ef8b2e1920e7\") " pod="opendatahub/model-serving-api-75bd89fdfc-h8bxb" Nov 02 17:35:01.014039 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.014018 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2fa7be0-9911-423c-adad-e88db47815a0-metrics-certs\") pod \"odh-model-controller-769875db56-27r8m\" (UID: \"f2fa7be0-9911-423c-adad-e88db47815a0\") " pod="opendatahub/odh-model-controller-769875db56-27r8m" Nov 02 17:35:01.014212 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.014191 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c7sz\" (UniqueName: \"kubernetes.io/projected/55697f5d-5f32-49d0-be8f-b616a718403f-kube-api-access-5c7sz\") pod \"llmisvc-controller-manager-869d5dcc5d-ht2xz\" (UID: \"55697f5d-5f32-49d0-be8f-b616a718403f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz" Nov 02 17:35:01.014265 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.014248 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2fa7be0-9911-423c-adad-e88db47815a0-cert\") pod \"odh-model-controller-769875db56-27r8m\" (UID: \"f2fa7be0-9911-423c-adad-e88db47815a0\") " pod="opendatahub/odh-model-controller-769875db56-27r8m" Nov 02 17:35:01.014331 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.014319 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/412157d1-0980-4139-bff7-466a1df1b0d3-cert\") pod \"kserve-controller-manager-776cc66d47-gt7zt\" (UID: \"412157d1-0980-4139-bff7-466a1df1b0d3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-gt7zt" Nov 02 17:35:01.014386 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.014374 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e543ec4-db22-4e57-bad4-ef8b2e1920e7-tls-certs\") pod \"model-serving-api-75bd89fdfc-h8bxb\" (UID: \"9e543ec4-db22-4e57-bad4-ef8b2e1920e7\") " pod="opendatahub/model-serving-api-75bd89fdfc-h8bxb" Nov 02 17:35:01.014433 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.014405 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55697f5d-5f32-49d0-be8f-b616a718403f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ht2xz\" (UID: \"55697f5d-5f32-49d0-be8f-b616a718403f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz" Nov 02 17:35:01.014519 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.014500 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxl8t\" (UniqueName: \"kubernetes.io/projected/412157d1-0980-4139-bff7-466a1df1b0d3-kube-api-access-lxl8t\") pod \"kserve-controller-manager-776cc66d47-gt7zt\" (UID: \"412157d1-0980-4139-bff7-466a1df1b0d3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-gt7zt" Nov 02 17:35:01.014985 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.014940 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" podStartSLOduration=2.3383097 podStartE2EDuration="14.014927763s" totalImagesPullingTime="11.676618063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:01.012410529 +0000 UTC m=+319.633770304" watchObservedRunningTime="2026-11-02 17:35:01.014927763 +0000 UTC m=+319.636287538" Nov 02 17:35:01.016768 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.016745 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/412157d1-0980-4139-bff7-466a1df1b0d3-cert\") pod \"kserve-controller-manager-776cc66d47-gt7zt\" (UID: \"412157d1-0980-4139-bff7-466a1df1b0d3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-gt7zt" Nov 02 17:35:01.039125 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.039097 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxl8t\" (UniqueName: \"kubernetes.io/projected/412157d1-0980-4139-bff7-466a1df1b0d3-kube-api-access-lxl8t\") pod \"kserve-controller-manager-776cc66d47-gt7zt\" (UID: \"412157d1-0980-4139-bff7-466a1df1b0d3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-gt7zt" Nov 02 17:35:01.069806 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.069775 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 02 17:35:01.069806 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.069792 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 02 17:35:01.070877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.070832 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-4xbwb\"" Nov 02 17:35:01.115294 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.115260 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2fa7be0-9911-423c-adad-e88db47815a0-metrics-certs\") pod \"odh-model-controller-769875db56-27r8m\" (UID: \"f2fa7be0-9911-423c-adad-e88db47815a0\") " pod="opendatahub/odh-model-controller-769875db56-27r8m" Nov 02 17:35:01.115451 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.115307 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c7sz\" (UniqueName: \"kubernetes.io/projected/55697f5d-5f32-49d0-be8f-b616a718403f-kube-api-access-5c7sz\") pod \"llmisvc-controller-manager-869d5dcc5d-ht2xz\" (UID: \"55697f5d-5f32-49d0-be8f-b616a718403f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz" Nov 02 17:35:01.115451 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.115364 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2fa7be0-9911-423c-adad-e88db47815a0-cert\") pod \"odh-model-controller-769875db56-27r8m\" (UID: \"f2fa7be0-9911-423c-adad-e88db47815a0\") " pod="opendatahub/odh-model-controller-769875db56-27r8m" Nov 02 17:35:01.115451 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.115440 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e543ec4-db22-4e57-bad4-ef8b2e1920e7-tls-certs\") pod \"model-serving-api-75bd89fdfc-h8bxb\" (UID: \"9e543ec4-db22-4e57-bad4-ef8b2e1920e7\") " pod="opendatahub/model-serving-api-75bd89fdfc-h8bxb" Nov 02 17:35:01.115624 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.115474 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55697f5d-5f32-49d0-be8f-b616a718403f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ht2xz\" (UID: \"55697f5d-5f32-49d0-be8f-b616a718403f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz" Nov 02 17:35:01.115624 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.115538 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tswzj\" (UniqueName: \"kubernetes.io/projected/f2fa7be0-9911-423c-adad-e88db47815a0-kube-api-access-tswzj\") pod \"odh-model-controller-769875db56-27r8m\" (UID: \"f2fa7be0-9911-423c-adad-e88db47815a0\") " pod="opendatahub/odh-model-controller-769875db56-27r8m" Nov 02 17:35:01.115624 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.115601 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf7lc\" (UniqueName: \"kubernetes.io/projected/9e543ec4-db22-4e57-bad4-ef8b2e1920e7-kube-api-access-zf7lc\") pod \"model-serving-api-75bd89fdfc-h8bxb\" (UID: \"9e543ec4-db22-4e57-bad4-ef8b2e1920e7\") " pod="opendatahub/model-serving-api-75bd89fdfc-h8bxb" Nov 02 17:35:01.118115 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.118089 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2fa7be0-9911-423c-adad-e88db47815a0-cert\") pod \"odh-model-controller-769875db56-27r8m\" (UID: \"f2fa7be0-9911-423c-adad-e88db47815a0\") " pod="opendatahub/odh-model-controller-769875db56-27r8m" Nov 02 17:35:01.118253 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.118134 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2fa7be0-9911-423c-adad-e88db47815a0-metrics-certs\") pod \"odh-model-controller-769875db56-27r8m\" (UID: \"f2fa7be0-9911-423c-adad-e88db47815a0\") " pod="opendatahub/odh-model-controller-769875db56-27r8m" Nov 02 17:35:01.118253 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.118156 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e543ec4-db22-4e57-bad4-ef8b2e1920e7-tls-certs\") pod \"model-serving-api-75bd89fdfc-h8bxb\" (UID: \"9e543ec4-db22-4e57-bad4-ef8b2e1920e7\") " pod="opendatahub/model-serving-api-75bd89fdfc-h8bxb" Nov 02 17:35:01.118253 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.118167 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55697f5d-5f32-49d0-be8f-b616a718403f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ht2xz\" (UID: \"55697f5d-5f32-49d0-be8f-b616a718403f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz" Nov 02 17:35:01.119953 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.119931 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-gt7zt" Nov 02 17:35:01.143826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.143789 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tswzj\" (UniqueName: \"kubernetes.io/projected/f2fa7be0-9911-423c-adad-e88db47815a0-kube-api-access-tswzj\") pod \"odh-model-controller-769875db56-27r8m\" (UID: \"f2fa7be0-9911-423c-adad-e88db47815a0\") " pod="opendatahub/odh-model-controller-769875db56-27r8m" Nov 02 17:35:01.143968 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.143875 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c7sz\" (UniqueName: \"kubernetes.io/projected/55697f5d-5f32-49d0-be8f-b616a718403f-kube-api-access-5c7sz\") pod \"llmisvc-controller-manager-869d5dcc5d-ht2xz\" (UID: \"55697f5d-5f32-49d0-be8f-b616a718403f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz" Nov 02 17:35:01.151858 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.151824 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf7lc\" (UniqueName: \"kubernetes.io/projected/9e543ec4-db22-4e57-bad4-ef8b2e1920e7-kube-api-access-zf7lc\") pod \"model-serving-api-75bd89fdfc-h8bxb\" (UID: \"9e543ec4-db22-4e57-bad4-ef8b2e1920e7\") " pod="opendatahub/model-serving-api-75bd89fdfc-h8bxb" Nov 02 17:35:01.192473 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.192442 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz" Nov 02 17:35:01.233329 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.233060 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-h8bxb" Nov 02 17:35:01.272237 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.272206 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-27r8m" Nov 02 17:35:01.420878 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.420816 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-gt7zt"] Nov 02 17:35:01.422399 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:35:01.422369 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412157d1_0980_4139_bff7_466a1df1b0d3.slice/crio-021c6f13cbf22758a46a0cce5bc311aa0fa8b94a204016861ef4ea166952efb0 WatchSource:0}: Error finding container 021c6f13cbf22758a46a0cce5bc311aa0fa8b94a204016861ef4ea166952efb0: Status 404 returned error can't find the container with id 021c6f13cbf22758a46a0cce5bc311aa0fa8b94a204016861ef4ea166952efb0 Nov 02 17:35:01.497044 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:35:01.497001 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod55697f5d_5f32_49d0_be8f_b616a718403f.slice/crio-e96d3f6faa97da20aebcc28c8ac42ec435b4382027027722f900a2a58420f41e WatchSource:0}: Error finding container e96d3f6faa97da20aebcc28c8ac42ec435b4382027027722f900a2a58420f41e: Status 404 returned error can't find the container with id e96d3f6faa97da20aebcc28c8ac42ec435b4382027027722f900a2a58420f41e Nov 02 17:35:01.498661 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.498637 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz"] Nov 02 17:35:01.544493 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.544468 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-h8bxb"] Nov 02 17:35:01.545805 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:35:01.545780 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e543ec4_db22_4e57_bad4_ef8b2e1920e7.slice/crio-e4f92ac86984b18b8c4c7ab2df82ab3c8f8cf3aede1814d69ed390a746916551 WatchSource:0}: Error finding container e4f92ac86984b18b8c4c7ab2df82ab3c8f8cf3aede1814d69ed390a746916551: Status 404 returned error can't find the container with id e4f92ac86984b18b8c4c7ab2df82ab3c8f8cf3aede1814d69ed390a746916551 Nov 02 17:35:01.577667 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.577645 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-27r8m"] Nov 02 17:35:01.578805 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:35:01.578744 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fa7be0_9911_423c_adad_e88db47815a0.slice/crio-70aea73a25a6b08c16b7459cdd6c8a73fed95926023b51ed55bc0a797b8dd0d8 WatchSource:0}: Error finding container 70aea73a25a6b08c16b7459cdd6c8a73fed95926023b51ed55bc0a797b8dd0d8: Status 404 returned error can't find the container with id 70aea73a25a6b08c16b7459cdd6c8a73fed95926023b51ed55bc0a797b8dd0d8 Nov 02 17:35:01.946305 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.946224 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz" event={"ID":"55697f5d-5f32-49d0-be8f-b616a718403f","Type":"ContainerStarted","Data":"e96d3f6faa97da20aebcc28c8ac42ec435b4382027027722f900a2a58420f41e"} Nov 02 17:35:01.947538 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.947509 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-h8bxb" event={"ID":"9e543ec4-db22-4e57-bad4-ef8b2e1920e7","Type":"ContainerStarted","Data":"e4f92ac86984b18b8c4c7ab2df82ab3c8f8cf3aede1814d69ed390a746916551"} Nov 02 17:35:01.948770 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.948746 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-gt7zt" event={"ID":"412157d1-0980-4139-bff7-466a1df1b0d3","Type":"ContainerStarted","Data":"021c6f13cbf22758a46a0cce5bc311aa0fa8b94a204016861ef4ea166952efb0"} Nov 02 17:35:01.949916 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:01.949894 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-27r8m" event={"ID":"f2fa7be0-9911-423c-adad-e88db47815a0","Type":"ContainerStarted","Data":"70aea73a25a6b08c16b7459cdd6c8a73fed95926023b51ed55bc0a797b8dd0d8"} Nov 02 17:35:05.993635 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:05.993607 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-h8bxb" event={"ID":"9e543ec4-db22-4e57-bad4-ef8b2e1920e7","Type":"ContainerStarted","Data":"1d71607b916f145f5ea2358a54da24f537db6c786ceaa01bb171cf0107450e03"} Nov 02 17:35:05.994032 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:05.993797 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-h8bxb" Nov 02 17:35:05.995390 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:05.995362 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-gt7zt" event={"ID":"412157d1-0980-4139-bff7-466a1df1b0d3","Type":"ContainerStarted","Data":"0b37456adc6708a32ab6586375885de97c1df428ed9ce802a97893a29a366219"} Nov 02 17:35:05.995499 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:05.995441 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-gt7zt" Nov 02 17:35:06.037101 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:06.036823 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-h8bxb" podStartSLOduration=2.556807169 podStartE2EDuration="6.036810536s" totalImagesPullingTime="3.480003367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:06.036264834 +0000 UTC m=+324.657624624" watchObservedRunningTime="2026-11-02 17:35:06.036810536 +0000 UTC m=+324.658170316" Nov 02 17:35:06.078266 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:06.078219 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-gt7zt" podStartSLOduration=1.671775534 podStartE2EDuration="6.078205183s" totalImagesPullingTime="4.406429649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:06.077080129 +0000 UTC m=+324.698439912" watchObservedRunningTime="2026-11-02 17:35:06.078205183 +0000 UTC m=+324.699564959" Nov 02 17:35:07.002622 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:07.002587 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-27r8m" event={"ID":"f2fa7be0-9911-423c-adad-e88db47815a0","Type":"ContainerStarted","Data":"c8bb9d5b3ecaa1eec23962624837d3c024ab202620f4dd6dead4f1a523e37d6f"} Nov 02 17:35:07.003140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:07.002667 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-27r8m" Nov 02 17:35:07.004330 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:07.004304 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz" event={"ID":"55697f5d-5f32-49d0-be8f-b616a718403f","Type":"ContainerStarted","Data":"42d5b1dd1c1b525cbc3ad7ea6a7f813c151a131eaf09efd376304e2b538554d9"} Nov 02 17:35:07.004596 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:07.004581 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz" Nov 02 17:35:07.047064 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:07.047005 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-27r8m" podStartSLOduration=2.659881054 podStartE2EDuration="7.046990581s" totalImagesPullingTime="4.387109527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:07.044102314 +0000 UTC m=+325.665462087" watchObservedRunningTime="2026-11-02 17:35:07.046990581 +0000 UTC m=+325.668350386" Nov 02 17:35:07.085790 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:07.085735 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz" podStartSLOduration=2.6212782629999998 podStartE2EDuration="7.085722119s" totalImagesPullingTime="4.464443856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:07.084645885 +0000 UTC m=+325.706005662" watchObservedRunningTime="2026-11-02 17:35:07.085722119 +0000 UTC m=+325.707081926" Nov 02 17:35:11.956511 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:11.956472 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bm8dm" Nov 02 17:35:17.009049 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:17.009013 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-h8bxb" Nov 02 17:35:18.012454 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:18.012422 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-27r8m" Nov 02 17:35:21.770997 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:35:21.770963 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:35:30.643142 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:30.643103 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-6hpp8"] Nov 02 17:35:30.803310 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:30.803263 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-6hpp8"] Nov 02 17:35:30.803501 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:30.803381 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-6hpp8" Nov 02 17:35:30.813843 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:30.813805 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qwkd\" (UniqueName: \"kubernetes.io/projected/6ecd5b82-6893-4191-8e64-719b5132ff39-kube-api-access-6qwkd\") pod \"authorino-6f6944bcdd-6hpp8\" (UID: \"6ecd5b82-6893-4191-8e64-719b5132ff39\") " pod="kuadrant-system/authorino-6f6944bcdd-6hpp8" Nov 02 17:35:30.857961 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:30.857920 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-6hpp8"] Nov 02 17:35:30.858203 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:35:30.858179 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6qwkd], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-6hpp8" podUID="6ecd5b82-6893-4191-8e64-719b5132ff39" Nov 02 17:35:30.915159 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:30.915061 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qwkd\" (UniqueName: \"kubernetes.io/projected/6ecd5b82-6893-4191-8e64-719b5132ff39-kube-api-access-6qwkd\") pod \"authorino-6f6944bcdd-6hpp8\" (UID: \"6ecd5b82-6893-4191-8e64-719b5132ff39\") " pod="kuadrant-system/authorino-6f6944bcdd-6hpp8" Nov 02 17:35:30.938996 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:30.938968 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qwkd\" (UniqueName: \"kubernetes.io/projected/6ecd5b82-6893-4191-8e64-719b5132ff39-kube-api-access-6qwkd\") pod \"authorino-6f6944bcdd-6hpp8\" (UID: \"6ecd5b82-6893-4191-8e64-719b5132ff39\") " pod="kuadrant-system/authorino-6f6944bcdd-6hpp8" Nov 02 17:35:31.064888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:31.064841 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-546c7b54c4-4m8g5"] Nov 02 17:35:31.091362 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:31.091324 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-546c7b54c4-4m8g5"] Nov 02 17:35:31.091523 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:31.091469 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-546c7b54c4-4m8g5" Nov 02 17:35:31.106935 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:31.106910 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 02 17:35:31.116318 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:31.116292 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnck6\" (UniqueName: \"kubernetes.io/projected/81a88e4d-d04c-41b8-99a1-f88fa6a7db6d-kube-api-access-cnck6\") pod \"authorino-546c7b54c4-4m8g5\" (UID: \"81a88e4d-d04c-41b8-99a1-f88fa6a7db6d\") " pod="kuadrant-system/authorino-546c7b54c4-4m8g5" Nov 02 17:35:31.116416 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:31.116340 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81a88e4d-d04c-41b8-99a1-f88fa6a7db6d-tls-cert\") pod \"authorino-546c7b54c4-4m8g5\" (UID: \"81a88e4d-d04c-41b8-99a1-f88fa6a7db6d\") " pod="kuadrant-system/authorino-546c7b54c4-4m8g5" Nov 02 17:35:31.144039 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:31.144006 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-6hpp8" Nov 02 17:35:31.165344 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:31.165286 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-6hpp8" Nov 02 17:35:31.217605 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:31.217578 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ecd5b82-6893-4191-8e64-719b5132ff39-kube-api-access-6qwkd\" (UniqueName: \"kubernetes.io/projected/6ecd5b82-6893-4191-8e64-719b5132ff39-kube-api-access-6qwkd\") pod \"6ecd5b82-6893-4191-8e64-719b5132ff39\" (UID: \"6ecd5b82-6893-4191-8e64-719b5132ff39\") " Nov 02 17:35:31.217749 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:31.217735 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnck6\" (UniqueName: \"kubernetes.io/projected/81a88e4d-d04c-41b8-99a1-f88fa6a7db6d-kube-api-access-cnck6\") pod \"authorino-546c7b54c4-4m8g5\" (UID: \"81a88e4d-d04c-41b8-99a1-f88fa6a7db6d\") " pod="kuadrant-system/authorino-546c7b54c4-4m8g5" Nov 02 17:35:31.217804 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:31.217779 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81a88e4d-d04c-41b8-99a1-f88fa6a7db6d-tls-cert\") pod \"authorino-546c7b54c4-4m8g5\" (UID: \"81a88e4d-d04c-41b8-99a1-f88fa6a7db6d\") " pod="kuadrant-system/authorino-546c7b54c4-4m8g5" Nov 02 17:35:31.219724 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:31.219692 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ecd5b82-6893-4191-8e64-719b5132ff39-kube-api-access-6qwkd" pod "6ecd5b82-6893-4191-8e64-719b5132ff39" (UID: "6ecd5b82-6893-4191-8e64-719b5132ff39"). InnerVolumeSpecName "kube-api-access-6qwkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:35:31.220347 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:31.220321 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81a88e4d-d04c-41b8-99a1-f88fa6a7db6d-tls-cert\") pod \"authorino-546c7b54c4-4m8g5\" (UID: \"81a88e4d-d04c-41b8-99a1-f88fa6a7db6d\") " pod="kuadrant-system/authorino-546c7b54c4-4m8g5" Nov 02 17:35:31.258433 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:31.258406 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnck6\" (UniqueName: \"kubernetes.io/projected/81a88e4d-d04c-41b8-99a1-f88fa6a7db6d-kube-api-access-cnck6\") pod \"authorino-546c7b54c4-4m8g5\" (UID: \"81a88e4d-d04c-41b8-99a1-f88fa6a7db6d\") " pod="kuadrant-system/authorino-546c7b54c4-4m8g5" Nov 02 17:35:31.318355 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:31.318327 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qwkd\" (UniqueName: \"kubernetes.io/projected/6ecd5b82-6893-4191-8e64-719b5132ff39-kube-api-access-6qwkd\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:35:31.418938 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:31.418834 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-546c7b54c4-4m8g5" Nov 02 17:35:31.648989 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:31.648952 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-v9l5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:35:31.649378 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:31.649006 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" podUID="62b05251-7074-41bc-8994-c1051abee475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:35:31.743788 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:31.743763 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-546c7b54c4-4m8g5"] Nov 02 17:35:31.745559 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:35:31.745527 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81a88e4d_d04c_41b8_99a1_f88fa6a7db6d.slice/crio-f10c02fd3fdaa19a9dd36ca5c5b8b81a9d2d0f4aa526114e0e4e2d132d73a9c8 WatchSource:0}: Error finding container f10c02fd3fdaa19a9dd36ca5c5b8b81a9d2d0f4aa526114e0e4e2d132d73a9c8: Status 404 returned error can't find the container with id f10c02fd3fdaa19a9dd36ca5c5b8b81a9d2d0f4aa526114e0e4e2d132d73a9c8 Nov 02 17:35:32.151073 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:32.150991 4860 generic.go:356] "Generic (PLEG): container finished" podID="62b05251-7074-41bc-8994-c1051abee475" containerID="afeda81a9e268826c31617880458f51bce447ac95759eea3a4e619b0cd6115a9" exitCode=0 Nov 02 17:35:32.151283 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:32.151077 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" event={"ID":"62b05251-7074-41bc-8994-c1051abee475","Type":"ContainerDied","Data":"afeda81a9e268826c31617880458f51bce447ac95759eea3a4e619b0cd6115a9"} Nov 02 17:35:32.151551 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:32.151521 4860 scope.go:122] "RemoveContainer" containerID="afeda81a9e268826c31617880458f51bce447ac95759eea3a4e619b0cd6115a9" Nov 02 17:35:32.152653 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:32.152625 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:35:32.152734 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:32.152699 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:35:32.152734 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:32.152707 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-6hpp8" Nov 02 17:35:32.152804 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:32.152734 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:35:32.152964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:32.152937 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-546c7b54c4-4m8g5" event={"ID":"81a88e4d-d04c-41b8-99a1-f88fa6a7db6d","Type":"ContainerStarted","Data":"f10c02fd3fdaa19a9dd36ca5c5b8b81a9d2d0f4aa526114e0e4e2d132d73a9c8"} Nov 02 17:35:32.262750 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:32.260432 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-6hpp8"] Nov 02 17:35:32.267878 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:32.267819 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-6hpp8"] Nov 02 17:35:33.195273 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:33.195218 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" event={"ID":"62b05251-7074-41bc-8994-c1051abee475","Type":"ContainerStarted","Data":"5a4a2440f0ea022d0c762cefdfcaa529ea8b44e559e079baffd8fd191a54c120"} Nov 02 17:35:33.197297 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:33.197263 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-546c7b54c4-4m8g5" event={"ID":"81a88e4d-d04c-41b8-99a1-f88fa6a7db6d","Type":"ContainerStarted","Data":"784a9b322b2da6da07d3262074e45da666785a2478b31cfb7d649118695b9d01"} Nov 02 17:35:33.308677 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:33.308615 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-546c7b54c4-4m8g5" podStartSLOduration=1.35416014 podStartE2EDuration="2.308597847s" totalImagesPullingTime="954.437707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:33.305812044 +0000 UTC m=+351.927171824" watchObservedRunningTime="2026-11-02 17:35:33.308597847 +0000 UTC m=+351.929957622" Nov 02 17:35:33.336389 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:33.336309 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5zwtt"] Nov 02 17:35:33.336666 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:33.336567 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-5zwtt" podUID="b792f0f9-8122-4134-a78b-b322dee30fd8" containerName="authorino" containerID="cri-o://3c71c6e07c66ad13e85e5dd875b987fea16b642eb37ec6546aee7cc2d515b083" gracePeriod=30 Nov 02 17:35:33.503266 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:33.503179 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ecd5b82-6893-4191-8e64-719b5132ff39" path="/var/lib/kubelet/pods/6ecd5b82-6893-4191-8e64-719b5132ff39/volumes" Nov 02 17:35:33.714038 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:33.714011 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5zwtt" Nov 02 17:35:33.748780 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:33.748754 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b792f0f9-8122-4134-a78b-b322dee30fd8-kube-api-access-q4nt8\" (UniqueName: \"kubernetes.io/projected/b792f0f9-8122-4134-a78b-b322dee30fd8-kube-api-access-q4nt8\") pod \"b792f0f9-8122-4134-a78b-b322dee30fd8\" (UID: \"b792f0f9-8122-4134-a78b-b322dee30fd8\") " Nov 02 17:35:33.750984 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:33.750953 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b792f0f9-8122-4134-a78b-b322dee30fd8-kube-api-access-q4nt8" pod "b792f0f9-8122-4134-a78b-b322dee30fd8" (UID: "b792f0f9-8122-4134-a78b-b322dee30fd8"). InnerVolumeSpecName "kube-api-access-q4nt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:35:33.849383 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:33.849346 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4nt8\" (UniqueName: \"kubernetes.io/projected/b792f0f9-8122-4134-a78b-b322dee30fd8-kube-api-access-q4nt8\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:35:34.047434 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:34.047352 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:35:34.047611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:34.047551 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-v9l5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Nov 02 17:35:34.047680 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:34.047603 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" podUID="62b05251-7074-41bc-8994-c1051abee475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Nov 02 17:35:34.204660 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:34.204621 4860 generic.go:356] "Generic (PLEG): container finished" podID="b792f0f9-8122-4134-a78b-b322dee30fd8" containerID="3c71c6e07c66ad13e85e5dd875b987fea16b642eb37ec6546aee7cc2d515b083" exitCode=0 Nov 02 17:35:34.205079 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:34.204684 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5zwtt" Nov 02 17:35:34.205079 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:34.204704 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5zwtt" event={"ID":"b792f0f9-8122-4134-a78b-b322dee30fd8","Type":"ContainerDied","Data":"3c71c6e07c66ad13e85e5dd875b987fea16b642eb37ec6546aee7cc2d515b083"} Nov 02 17:35:34.205079 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:34.204757 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5zwtt" event={"ID":"b792f0f9-8122-4134-a78b-b322dee30fd8","Type":"ContainerDied","Data":"cbc1a2f14cee7c23288d8b2238bfc360ecaf5d25abe45770af04802aed2782a1"} Nov 02 17:35:34.205079 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:34.204774 4860 scope.go:122] "RemoveContainer" containerID="3c71c6e07c66ad13e85e5dd875b987fea16b642eb37ec6546aee7cc2d515b083" Nov 02 17:35:34.214604 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:34.214585 4860 scope.go:122] "RemoveContainer" containerID="3c71c6e07c66ad13e85e5dd875b987fea16b642eb37ec6546aee7cc2d515b083" Nov 02 17:35:34.214924 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:35:34.214903 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c71c6e07c66ad13e85e5dd875b987fea16b642eb37ec6546aee7cc2d515b083\": container with ID starting with 3c71c6e07c66ad13e85e5dd875b987fea16b642eb37ec6546aee7cc2d515b083 not found: ID does not exist" containerID="3c71c6e07c66ad13e85e5dd875b987fea16b642eb37ec6546aee7cc2d515b083" Nov 02 17:35:34.214986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:34.214933 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c71c6e07c66ad13e85e5dd875b987fea16b642eb37ec6546aee7cc2d515b083"} err="failed to get container status \"3c71c6e07c66ad13e85e5dd875b987fea16b642eb37ec6546aee7cc2d515b083\": rpc error: code = NotFound desc = could not find container \"3c71c6e07c66ad13e85e5dd875b987fea16b642eb37ec6546aee7cc2d515b083\": container with ID starting with 3c71c6e07c66ad13e85e5dd875b987fea16b642eb37ec6546aee7cc2d515b083 not found: ID does not exist" Nov 02 17:35:34.280245 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:34.280096 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5zwtt"] Nov 02 17:35:34.282455 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:34.282431 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5zwtt"] Nov 02 17:35:35.047593 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:35.047560 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-v9l5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Nov 02 17:35:35.047760 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:35.047627 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" podUID="62b05251-7074-41bc-8994-c1051abee475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Nov 02 17:35:35.497665 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:35.497629 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b792f0f9-8122-4134-a78b-b322dee30fd8" path="/var/lib/kubelet/pods/b792f0f9-8122-4134-a78b-b322dee30fd8/volumes" Nov 02 17:35:36.048051 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:36.048011 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-v9l5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Nov 02 17:35:36.048223 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:36.048079 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" podUID="62b05251-7074-41bc-8994-c1051abee475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Nov 02 17:35:37.010086 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:37.010060 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-gt7zt" Nov 02 17:35:37.048427 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:37.048392 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-v9l5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Nov 02 17:35:37.048583 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:37.048456 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" podUID="62b05251-7074-41bc-8994-c1051abee475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Nov 02 17:35:38.013113 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:38.013082 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz" Nov 02 17:35:38.047819 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:38.047790 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-v9l5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Nov 02 17:35:38.048009 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:38.047838 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" podUID="62b05251-7074-41bc-8994-c1051abee475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Nov 02 17:35:39.051074 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:39.051031 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:35:39.051559 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:39.051423 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:35:39.052275 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:39.052258 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 17:35:51.769259 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:35:51.769223 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:35:54.529074 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:54.529033 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fdn9v"] Nov 02 17:35:54.529570 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:54.529550 4860 cpu_manager.go:414] "containerMap: removing container" podUID="b792f0f9-8122-4134-a78b-b322dee30fd8" containerName="authorino" Nov 02 17:35:54.529613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:54.529572 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b792f0f9-8122-4134-a78b-b322dee30fd8" containerName="authorino" Nov 02 17:35:54.529675 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:54.529665 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b792f0f9-8122-4134-a78b-b322dee30fd8" containerName="authorino" Nov 02 17:35:54.531791 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:54.531775 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fdn9v" Nov 02 17:35:54.554428 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:54.554406 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fdn9v"] Nov 02 17:35:54.659110 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:54.659080 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8ee3d1e-c5bb-4c46-977e-cb4b2d183266-cert\") pod \"llmisvc-controller-manager-6466c554d8-fdn9v\" (UID: \"e8ee3d1e-c5bb-4c46-977e-cb4b2d183266\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fdn9v" Nov 02 17:35:54.659280 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:54.659127 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29xlv\" (UniqueName: \"kubernetes.io/projected/e8ee3d1e-c5bb-4c46-977e-cb4b2d183266-kube-api-access-29xlv\") pod \"llmisvc-controller-manager-6466c554d8-fdn9v\" (UID: \"e8ee3d1e-c5bb-4c46-977e-cb4b2d183266\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fdn9v" Nov 02 17:35:54.759810 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:54.759767 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29xlv\" (UniqueName: \"kubernetes.io/projected/e8ee3d1e-c5bb-4c46-977e-cb4b2d183266-kube-api-access-29xlv\") pod \"llmisvc-controller-manager-6466c554d8-fdn9v\" (UID: \"e8ee3d1e-c5bb-4c46-977e-cb4b2d183266\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fdn9v" Nov 02 17:35:54.759993 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:54.759913 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8ee3d1e-c5bb-4c46-977e-cb4b2d183266-cert\") pod \"llmisvc-controller-manager-6466c554d8-fdn9v\" (UID: \"e8ee3d1e-c5bb-4c46-977e-cb4b2d183266\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fdn9v" Nov 02 17:35:54.762122 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:54.762100 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8ee3d1e-c5bb-4c46-977e-cb4b2d183266-cert\") pod \"llmisvc-controller-manager-6466c554d8-fdn9v\" (UID: \"e8ee3d1e-c5bb-4c46-977e-cb4b2d183266\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fdn9v" Nov 02 17:35:54.783893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:54.783833 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29xlv\" (UniqueName: \"kubernetes.io/projected/e8ee3d1e-c5bb-4c46-977e-cb4b2d183266-kube-api-access-29xlv\") pod \"llmisvc-controller-manager-6466c554d8-fdn9v\" (UID: \"e8ee3d1e-c5bb-4c46-977e-cb4b2d183266\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fdn9v" Nov 02 17:35:54.843673 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:54.843650 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fdn9v" Nov 02 17:35:55.140119 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:55.140087 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fdn9v"] Nov 02 17:35:55.141826 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:35:55.141792 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ee3d1e_c5bb_4c46_977e_cb4b2d183266.slice/crio-14492182b4054b5642b605a6e944362eb18b9867a006664bd0d21805a3cbc659 WatchSource:0}: Error finding container 14492182b4054b5642b605a6e944362eb18b9867a006664bd0d21805a3cbc659: Status 404 returned error can't find the container with id 14492182b4054b5642b605a6e944362eb18b9867a006664bd0d21805a3cbc659 Nov 02 17:35:55.329869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:55.329811 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fdn9v" event={"ID":"e8ee3d1e-c5bb-4c46-977e-cb4b2d183266","Type":"ContainerStarted","Data":"14492182b4054b5642b605a6e944362eb18b9867a006664bd0d21805a3cbc659"} Nov 02 17:35:56.337379 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:56.337338 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fdn9v" event={"ID":"e8ee3d1e-c5bb-4c46-977e-cb4b2d183266","Type":"ContainerStarted","Data":"160809fc8166f2930007fd49c5bc1c1153b93bba812128927fd11f7f447e6c0b"} Nov 02 17:35:56.337763 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:56.337400 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fdn9v" Nov 02 17:35:56.380743 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:35:56.380687 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fdn9v" podStartSLOduration=2.02801576 podStartE2EDuration="2.380674914s" totalImagesPullingTime="352.659154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:56.378143807 +0000 UTC m=+374.999503583" watchObservedRunningTime="2026-11-02 17:35:56.380674914 +0000 UTC m=+375.002034689" Nov 02 17:36:21.768684 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:36:21.768648 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:36:27.345379 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:27.345347 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fdn9v" Nov 02 17:36:27.419126 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:27.419091 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz"] Nov 02 17:36:27.419331 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:27.419312 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz" podUID="55697f5d-5f32-49d0-be8f-b616a718403f" containerName="manager" containerID="cri-o://42d5b1dd1c1b525cbc3ad7ea6a7f813c151a131eaf09efd376304e2b538554d9" gracePeriod=30 Nov 02 17:36:27.668528 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:27.668506 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz" Nov 02 17:36:27.747161 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:27.747125 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55697f5d-5f32-49d0-be8f-b616a718403f-kube-api-access-5c7sz\" (UniqueName: \"kubernetes.io/projected/55697f5d-5f32-49d0-be8f-b616a718403f-kube-api-access-5c7sz\") pod \"55697f5d-5f32-49d0-be8f-b616a718403f\" (UID: \"55697f5d-5f32-49d0-be8f-b616a718403f\") " Nov 02 17:36:27.747161 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:27.747166 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55697f5d-5f32-49d0-be8f-b616a718403f-cert\" (UniqueName: \"kubernetes.io/secret/55697f5d-5f32-49d0-be8f-b616a718403f-cert\") pod \"55697f5d-5f32-49d0-be8f-b616a718403f\" (UID: \"55697f5d-5f32-49d0-be8f-b616a718403f\") " Nov 02 17:36:27.749290 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:27.749257 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55697f5d-5f32-49d0-be8f-b616a718403f-kube-api-access-5c7sz" pod "55697f5d-5f32-49d0-be8f-b616a718403f" (UID: "55697f5d-5f32-49d0-be8f-b616a718403f"). InnerVolumeSpecName "kube-api-access-5c7sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:36:27.749290 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:27.749281 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55697f5d-5f32-49d0-be8f-b616a718403f-cert" pod "55697f5d-5f32-49d0-be8f-b616a718403f" (UID: "55697f5d-5f32-49d0-be8f-b616a718403f"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:36:27.848321 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:27.848292 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c7sz\" (UniqueName: \"kubernetes.io/projected/55697f5d-5f32-49d0-be8f-b616a718403f-kube-api-access-5c7sz\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:36:27.848321 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:27.848320 4860 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55697f5d-5f32-49d0-be8f-b616a718403f-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:36:28.521286 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:28.521252 4860 generic.go:356] "Generic (PLEG): container finished" podID="55697f5d-5f32-49d0-be8f-b616a718403f" containerID="42d5b1dd1c1b525cbc3ad7ea6a7f813c151a131eaf09efd376304e2b538554d9" exitCode=0 Nov 02 17:36:28.521678 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:28.521300 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz" Nov 02 17:36:28.521678 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:28.521338 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz" event={"ID":"55697f5d-5f32-49d0-be8f-b616a718403f","Type":"ContainerDied","Data":"42d5b1dd1c1b525cbc3ad7ea6a7f813c151a131eaf09efd376304e2b538554d9"} Nov 02 17:36:28.521678 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:28.521396 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz" event={"ID":"55697f5d-5f32-49d0-be8f-b616a718403f","Type":"ContainerDied","Data":"e96d3f6faa97da20aebcc28c8ac42ec435b4382027027722f900a2a58420f41e"} Nov 02 17:36:28.521678 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:28.521417 4860 scope.go:122] "RemoveContainer" containerID="42d5b1dd1c1b525cbc3ad7ea6a7f813c151a131eaf09efd376304e2b538554d9" Nov 02 17:36:28.535005 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:28.534981 4860 scope.go:122] "RemoveContainer" containerID="42d5b1dd1c1b525cbc3ad7ea6a7f813c151a131eaf09efd376304e2b538554d9" Nov 02 17:36:28.535361 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:36:28.535334 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42d5b1dd1c1b525cbc3ad7ea6a7f813c151a131eaf09efd376304e2b538554d9\": container with ID starting with 42d5b1dd1c1b525cbc3ad7ea6a7f813c151a131eaf09efd376304e2b538554d9 not found: ID does not exist" containerID="42d5b1dd1c1b525cbc3ad7ea6a7f813c151a131eaf09efd376304e2b538554d9" Nov 02 17:36:28.535438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:28.535374 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42d5b1dd1c1b525cbc3ad7ea6a7f813c151a131eaf09efd376304e2b538554d9"} err="failed to get container status \"42d5b1dd1c1b525cbc3ad7ea6a7f813c151a131eaf09efd376304e2b538554d9\": rpc error: code = NotFound desc = could not find container \"42d5b1dd1c1b525cbc3ad7ea6a7f813c151a131eaf09efd376304e2b538554d9\": container with ID starting with 42d5b1dd1c1b525cbc3ad7ea6a7f813c151a131eaf09efd376304e2b538554d9 not found: ID does not exist" Nov 02 17:36:28.585075 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:28.585051 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz"] Nov 02 17:36:28.588752 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:28.588733 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ht2xz"] Nov 02 17:36:29.496974 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:36:29.496945 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55697f5d-5f32-49d0-be8f-b616a718403f" path="/var/lib/kubelet/pods/55697f5d-5f32-49d0-be8f-b616a718403f/volumes" Nov 02 17:36:51.777885 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:36:51.777828 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:37:15.696030 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:15.695994 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-546c7b54c4-4m8g5"] Nov 02 17:37:15.696664 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:15.696194 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-546c7b54c4-4m8g5" podUID="81a88e4d-d04c-41b8-99a1-f88fa6a7db6d" containerName="authorino" containerID="cri-o://784a9b322b2da6da07d3262074e45da666785a2478b31cfb7d649118695b9d01" gracePeriod=30 Nov 02 17:37:15.829738 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:15.829704 4860 generic.go:356] "Generic (PLEG): container finished" podID="81a88e4d-d04c-41b8-99a1-f88fa6a7db6d" containerID="784a9b322b2da6da07d3262074e45da666785a2478b31cfb7d649118695b9d01" exitCode=0 Nov 02 17:37:15.829928 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:15.829765 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-546c7b54c4-4m8g5" event={"ID":"81a88e4d-d04c-41b8-99a1-f88fa6a7db6d","Type":"ContainerDied","Data":"784a9b322b2da6da07d3262074e45da666785a2478b31cfb7d649118695b9d01"} Nov 02 17:37:15.959207 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:15.959152 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-546c7b54c4-4m8g5" Nov 02 17:37:16.017545 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:16.017513 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81a88e4d-d04c-41b8-99a1-f88fa6a7db6d-tls-cert\" (UniqueName: \"kubernetes.io/secret/81a88e4d-d04c-41b8-99a1-f88fa6a7db6d-tls-cert\") pod \"81a88e4d-d04c-41b8-99a1-f88fa6a7db6d\" (UID: \"81a88e4d-d04c-41b8-99a1-f88fa6a7db6d\") " Nov 02 17:37:16.017709 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:16.017686 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a88e4d-d04c-41b8-99a1-f88fa6a7db6d-kube-api-access-cnck6\" (UniqueName: \"kubernetes.io/projected/81a88e4d-d04c-41b8-99a1-f88fa6a7db6d-kube-api-access-cnck6\") pod \"81a88e4d-d04c-41b8-99a1-f88fa6a7db6d\" (UID: \"81a88e4d-d04c-41b8-99a1-f88fa6a7db6d\") " Nov 02 17:37:16.019701 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:16.019669 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a88e4d-d04c-41b8-99a1-f88fa6a7db6d-kube-api-access-cnck6" pod "81a88e4d-d04c-41b8-99a1-f88fa6a7db6d" (UID: "81a88e4d-d04c-41b8-99a1-f88fa6a7db6d"). InnerVolumeSpecName "kube-api-access-cnck6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:37:16.028642 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:16.028610 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81a88e4d-d04c-41b8-99a1-f88fa6a7db6d-tls-cert" pod "81a88e4d-d04c-41b8-99a1-f88fa6a7db6d" (UID: "81a88e4d-d04c-41b8-99a1-f88fa6a7db6d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:37:16.119359 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:16.119324 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnck6\" (UniqueName: \"kubernetes.io/projected/81a88e4d-d04c-41b8-99a1-f88fa6a7db6d-kube-api-access-cnck6\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:37:16.119359 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:16.119353 4860 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81a88e4d-d04c-41b8-99a1-f88fa6a7db6d-tls-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:37:16.836935 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:16.836905 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-546c7b54c4-4m8g5" Nov 02 17:37:16.836935 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:16.836924 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-546c7b54c4-4m8g5" event={"ID":"81a88e4d-d04c-41b8-99a1-f88fa6a7db6d","Type":"ContainerDied","Data":"f10c02fd3fdaa19a9dd36ca5c5b8b81a9d2d0f4aa526114e0e4e2d132d73a9c8"} Nov 02 17:37:16.837409 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:16.836976 4860 scope.go:122] "RemoveContainer" containerID="784a9b322b2da6da07d3262074e45da666785a2478b31cfb7d649118695b9d01" Nov 02 17:37:16.891572 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:16.891543 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-546c7b54c4-4m8g5"] Nov 02 17:37:16.904297 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:16.904273 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-546c7b54c4-4m8g5"] Nov 02 17:37:17.498355 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:17.498324 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a88e4d-d04c-41b8-99a1-f88fa6a7db6d" path="/var/lib/kubelet/pods/81a88e4d-d04c-41b8-99a1-f88fa6a7db6d/volumes" Nov 02 17:37:21.773085 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:37:21.773041 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:37:47.847385 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:47.847348 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb"] Nov 02 17:37:47.847913 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:47.847889 4860 cpu_manager.go:414] "containerMap: removing container" podUID="55697f5d-5f32-49d0-be8f-b616a718403f" containerName="manager" Nov 02 17:37:47.847913 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:47.847912 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55697f5d-5f32-49d0-be8f-b616a718403f" containerName="manager" Nov 02 17:37:47.848010 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:47.847948 4860 cpu_manager.go:414] "containerMap: removing container" podUID="81a88e4d-d04c-41b8-99a1-f88fa6a7db6d" containerName="authorino" Nov 02 17:37:47.848010 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:47.847954 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a88e4d-d04c-41b8-99a1-f88fa6a7db6d" containerName="authorino" Nov 02 17:37:47.848085 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:47.848027 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a88e4d-d04c-41b8-99a1-f88fa6a7db6d" containerName="authorino" Nov 02 17:37:47.848085 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:47.848036 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="55697f5d-5f32-49d0-be8f-b616a718403f" containerName="manager" Nov 02 17:37:47.852312 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:47.852295 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:47.887357 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:47.887326 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb"] Nov 02 17:37:48.006888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.006833 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.007062 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.006904 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.007062 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.006982 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.007150 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.007058 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.007150 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.007093 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.007150 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.007114 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.007281 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.007173 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.007281 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.007252 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr22w\" (UniqueName: \"kubernetes.io/projected/6d6284d8-6036-4088-a7e0-a1d902f27ecb-kube-api-access-zr22w\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.007396 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.007331 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.108015 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.107931 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.108015 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.107972 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.108015 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.107994 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.108015 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.108012 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.108338 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.108031 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr22w\" (UniqueName: \"kubernetes.io/projected/6d6284d8-6036-4088-a7e0-a1d902f27ecb-kube-api-access-zr22w\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.108338 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.108047 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.108338 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.108087 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.108338 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.108128 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.108338 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.108231 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.108598 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.108463 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.108655 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.108591 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.108716 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.108666 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.108716 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.108682 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.108937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.108914 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.110264 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.110242 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.110575 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.110556 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.131691 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.131664 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.131876 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.131839 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr22w\" (UniqueName: \"kubernetes.io/projected/6d6284d8-6036-4088-a7e0-a1d902f27ecb-kube-api-access-zr22w\") pod \"maas-default-gateway-openshift-default-76f4876667-qlptb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.166462 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.166434 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:48.480294 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.480265 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb"] Nov 02 17:37:48.481944 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:37:48.481916 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6284d8_6036_4088_a7e0_a1d902f27ecb.slice/crio-114bb94bc3ec337561879b83ef25841f1de516c9bfc04d9da613e63dbabbe44c WatchSource:0}: Error finding container 114bb94bc3ec337561879b83ef25841f1de516c9bfc04d9da613e63dbabbe44c: Status 404 returned error can't find the container with id 114bb94bc3ec337561879b83ef25841f1de516c9bfc04d9da613e63dbabbe44c Nov 02 17:37:48.484308 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.484273 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:37:48.484392 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.484346 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:37:48.484392 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:48.484374 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:37:49.016683 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:49.016621 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" event={"ID":"6d6284d8-6036-4088-a7e0-a1d902f27ecb","Type":"ContainerStarted","Data":"589d2ccd71935caf23ff4b0c79740268505c3699ba457eca5c3fa94cb67c335c"} Nov 02 17:37:49.016683 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:49.016673 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" event={"ID":"6d6284d8-6036-4088-a7e0-a1d902f27ecb","Type":"ContainerStarted","Data":"114bb94bc3ec337561879b83ef25841f1de516c9bfc04d9da613e63dbabbe44c"} Nov 02 17:37:49.081053 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:49.080984 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" podStartSLOduration=2.080965411 podStartE2EDuration="2.080965411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:49.077410997 +0000 UTC m=+487.698770812" watchObservedRunningTime="2026-11-02 17:37:49.080965411 +0000 UTC m=+487.702325181" Nov 02 17:37:49.168717 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:49.168663 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-qlptb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:37:49.169065 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:49.169020 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" podUID="6d6284d8-6036-4088-a7e0-a1d902f27ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 02 17:37:49.169530 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:49.169513 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:50.174309 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:50.174267 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-qlptb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:37:50.174915 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:50.174881 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" podUID="6d6284d8-6036-4088-a7e0-a1d902f27ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 02 17:37:51.173247 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:51.173206 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-qlptb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:37:51.173521 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:51.173491 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" podUID="6d6284d8-6036-4088-a7e0-a1d902f27ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 02 17:37:51.795386 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:37:51.795349 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:37:52.173872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:52.173282 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-qlptb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:37:52.173872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:52.173346 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" podUID="6d6284d8-6036-4088-a7e0-a1d902f27ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 02 17:37:53.173325 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:53.173237 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-qlptb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:37:53.174013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:53.173981 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" podUID="6d6284d8-6036-4088-a7e0-a1d902f27ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 02 17:37:54.174873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:54.171317 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-qlptb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:37:54.174873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:54.171387 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" podUID="6d6284d8-6036-4088-a7e0-a1d902f27ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 02 17:37:55.167357 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:55.167312 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-qlptb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 02 17:37:55.167567 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:55.167369 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" podUID="6d6284d8-6036-4088-a7e0-a1d902f27ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 02 17:37:56.685152 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:56.685112 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:56.685663 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:56.685524 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:37:56.686255 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:37:56.686231 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:38:21.249240 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.249203 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj"] Nov 02 17:38:21.253122 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.253104 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" Nov 02 17:38:21.281160 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.281134 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-120394\"" Nov 02 17:38:21.281346 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.281123 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-120394-dockercfg-lh76t\"" Nov 02 17:38:21.283859 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.283825 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj"] Nov 02 17:38:21.323278 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.323243 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnmk8\" (UniqueName: \"kubernetes.io/projected/75e58f61-43a2-4600-a380-549625283343-kube-api-access-lnmk8\") pod \"payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj\" (UID: \"75e58f61-43a2-4600-a380-549625283343\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" Nov 02 17:38:21.323428 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.323399 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e58f61-43a2-4600-a380-549625283343-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj\" (UID: \"75e58f61-43a2-4600-a380-549625283343\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" Nov 02 17:38:21.369130 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.369098 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk"] Nov 02 17:38:21.373131 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.373103 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" Nov 02 17:38:21.395651 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.395618 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk"] Nov 02 17:38:21.424473 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.424432 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e58f61-43a2-4600-a380-549625283343-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj\" (UID: \"75e58f61-43a2-4600-a380-549625283343\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" Nov 02 17:38:21.424473 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.424480 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmk8\" (UniqueName: \"kubernetes.io/projected/75e58f61-43a2-4600-a380-549625283343-kube-api-access-lnmk8\") pod \"payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj\" (UID: \"75e58f61-43a2-4600-a380-549625283343\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" Nov 02 17:38:21.424712 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.424511 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xmv9\" (UniqueName: \"kubernetes.io/projected/bafb2593-6516-446c-b113-37bff6320729-kube-api-access-5xmv9\") pod \"payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk\" (UID: \"bafb2593-6516-446c-b113-37bff6320729\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" Nov 02 17:38:21.424712 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.424554 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bafb2593-6516-446c-b113-37bff6320729-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk\" (UID: \"bafb2593-6516-446c-b113-37bff6320729\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" Nov 02 17:38:21.425337 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.425308 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e58f61-43a2-4600-a380-549625283343-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj\" (UID: \"75e58f61-43a2-4600-a380-549625283343\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" Nov 02 17:38:21.449087 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.449059 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnmk8\" (UniqueName: \"kubernetes.io/projected/75e58f61-43a2-4600-a380-549625283343-kube-api-access-lnmk8\") pod \"payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj\" (UID: \"75e58f61-43a2-4600-a380-549625283343\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" Nov 02 17:38:21.525386 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.525301 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xmv9\" (UniqueName: \"kubernetes.io/projected/bafb2593-6516-446c-b113-37bff6320729-kube-api-access-5xmv9\") pod \"payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk\" (UID: \"bafb2593-6516-446c-b113-37bff6320729\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" Nov 02 17:38:21.525386 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.525355 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bafb2593-6516-446c-b113-37bff6320729-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk\" (UID: \"bafb2593-6516-446c-b113-37bff6320729\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" Nov 02 17:38:21.526074 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.526054 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bafb2593-6516-446c-b113-37bff6320729-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk\" (UID: \"bafb2593-6516-446c-b113-37bff6320729\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" Nov 02 17:38:21.557015 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.556988 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xmv9\" (UniqueName: \"kubernetes.io/projected/bafb2593-6516-446c-b113-37bff6320729-kube-api-access-5xmv9\") pod \"payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk\" (UID: \"bafb2593-6516-446c-b113-37bff6320729\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" Nov 02 17:38:21.566791 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.566761 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" Nov 02 17:38:21.687300 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.687268 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" Nov 02 17:38:21.783101 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:38:21.783011 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:38:21.882723 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:21.882695 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj"] Nov 02 17:38:21.884164 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:38:21.884139 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e58f61_43a2_4600_a380_549625283343.slice/crio-50c1bb31829e756c31b1c5f1e3d29e8e3fc3d232c71556254908bb6dcc320635 WatchSource:0}: Error finding container 50c1bb31829e756c31b1c5f1e3d29e8e3fc3d232c71556254908bb6dcc320635: Status 404 returned error can't find the container with id 50c1bb31829e756c31b1c5f1e3d29e8e3fc3d232c71556254908bb6dcc320635 Nov 02 17:38:22.001037 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:22.001006 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk"] Nov 02 17:38:22.001915 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:38:22.001889 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbafb2593_6516_446c_b113_37bff6320729.slice/crio-b38a71891f2d5ad74cc0509065a050a0152fce42064f454449f75787f9c52bae WatchSource:0}: Error finding container b38a71891f2d5ad74cc0509065a050a0152fce42064f454449f75787f9c52bae: Status 404 returned error can't find the container with id b38a71891f2d5ad74cc0509065a050a0152fce42064f454449f75787f9c52bae Nov 02 17:38:22.217348 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:22.217313 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" event={"ID":"75e58f61-43a2-4600-a380-549625283343","Type":"ContainerStarted","Data":"50c1bb31829e756c31b1c5f1e3d29e8e3fc3d232c71556254908bb6dcc320635"} Nov 02 17:38:22.218591 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:22.218559 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" event={"ID":"bafb2593-6516-446c-b113-37bff6320729","Type":"ContainerStarted","Data":"b38a71891f2d5ad74cc0509065a050a0152fce42064f454449f75787f9c52bae"} Nov 02 17:38:23.058587 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.058535 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl"] Nov 02 17:38:23.062677 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.062653 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl" Nov 02 17:38:23.078330 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.078302 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-7df526\"" Nov 02 17:38:23.090891 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.090724 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-7df526-dockercfg-bbz5x\"" Nov 02 17:38:23.096201 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.095148 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl"] Nov 02 17:38:23.152874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.148724 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63359d54-2671-4cbc-9033-cc1716d65580-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl\" (UID: \"63359d54-2671-4cbc-9033-cc1716d65580\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl" Nov 02 17:38:23.152874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.148985 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzlbq\" (UniqueName: \"kubernetes.io/projected/63359d54-2671-4cbc-9033-cc1716d65580-kube-api-access-hzlbq\") pod \"payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl\" (UID: \"63359d54-2671-4cbc-9033-cc1716d65580\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl" Nov 02 17:38:23.160890 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.157925 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm"] Nov 02 17:38:23.162458 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.162421 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm" Nov 02 17:38:23.175169 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.175140 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm"] Nov 02 17:38:23.249915 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.249875 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/980e42a5-a602-46ea-842d-6b6914e1b870-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm\" (UID: \"980e42a5-a602-46ea-842d-6b6914e1b870\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm" Nov 02 17:38:23.250099 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.249938 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj5vc\" (UniqueName: \"kubernetes.io/projected/980e42a5-a602-46ea-842d-6b6914e1b870-kube-api-access-zj5vc\") pod \"payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm\" (UID: \"980e42a5-a602-46ea-842d-6b6914e1b870\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm" Nov 02 17:38:23.250099 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.250076 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63359d54-2671-4cbc-9033-cc1716d65580-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl\" (UID: \"63359d54-2671-4cbc-9033-cc1716d65580\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl" Nov 02 17:38:23.250218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.250155 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzlbq\" (UniqueName: \"kubernetes.io/projected/63359d54-2671-4cbc-9033-cc1716d65580-kube-api-access-hzlbq\") pod \"payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl\" (UID: \"63359d54-2671-4cbc-9033-cc1716d65580\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl" Nov 02 17:38:23.250985 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.250965 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63359d54-2671-4cbc-9033-cc1716d65580-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl\" (UID: \"63359d54-2671-4cbc-9033-cc1716d65580\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl" Nov 02 17:38:23.275614 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.275580 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzlbq\" (UniqueName: \"kubernetes.io/projected/63359d54-2671-4cbc-9033-cc1716d65580-kube-api-access-hzlbq\") pod \"payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl\" (UID: \"63359d54-2671-4cbc-9033-cc1716d65580\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl" Nov 02 17:38:23.351439 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.351353 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/980e42a5-a602-46ea-842d-6b6914e1b870-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm\" (UID: \"980e42a5-a602-46ea-842d-6b6914e1b870\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm" Nov 02 17:38:23.351439 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.351412 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj5vc\" (UniqueName: \"kubernetes.io/projected/980e42a5-a602-46ea-842d-6b6914e1b870-kube-api-access-zj5vc\") pod \"payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm\" (UID: \"980e42a5-a602-46ea-842d-6b6914e1b870\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm" Nov 02 17:38:23.352374 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.352339 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/980e42a5-a602-46ea-842d-6b6914e1b870-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm\" (UID: \"980e42a5-a602-46ea-842d-6b6914e1b870\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm" Nov 02 17:38:23.375019 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.374989 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj5vc\" (UniqueName: \"kubernetes.io/projected/980e42a5-a602-46ea-842d-6b6914e1b870-kube-api-access-zj5vc\") pod \"payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm\" (UID: \"980e42a5-a602-46ea-842d-6b6914e1b870\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm" Nov 02 17:38:23.382026 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.381816 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl" Nov 02 17:38:23.484360 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.484326 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm" Nov 02 17:38:23.699355 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.699326 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl"] Nov 02 17:38:23.700388 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:38:23.700353 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63359d54_2671_4cbc_9033_cc1716d65580.slice/crio-d7fdc8a53ff9c5acb20d3de679575b9f4334fbe267be6b58652ee6fecaf0cf24 WatchSource:0}: Error finding container d7fdc8a53ff9c5acb20d3de679575b9f4334fbe267be6b58652ee6fecaf0cf24: Status 404 returned error can't find the container with id d7fdc8a53ff9c5acb20d3de679575b9f4334fbe267be6b58652ee6fecaf0cf24 Nov 02 17:38:23.813423 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.813391 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm"] Nov 02 17:38:23.814501 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:38:23.814472 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980e42a5_a602_46ea_842d_6b6914e1b870.slice/crio-a7b8f0a5793605ea4afdc92ff2696051117f90e0996b4e42c7a7e77e21a960de WatchSource:0}: Error finding container a7b8f0a5793605ea4afdc92ff2696051117f90e0996b4e42c7a7e77e21a960de: Status 404 returned error can't find the container with id a7b8f0a5793605ea4afdc92ff2696051117f90e0996b4e42c7a7e77e21a960de Nov 02 17:38:23.872407 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.872364 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76"] Nov 02 17:38:23.883383 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.883353 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76" Nov 02 17:38:23.904569 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.904543 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-5eedab-dockercfg-ddp5d\"" Nov 02 17:38:23.909519 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.909489 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76"] Nov 02 17:38:23.910056 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.910034 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-5eedab\"" Nov 02 17:38:23.963620 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.963512 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsbk5\" (UniqueName: \"kubernetes.io/projected/caee9964-de70-4471-9013-90b120d20fa1-kube-api-access-fsbk5\") pod \"payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76\" (UID: \"caee9964-de70-4471-9013-90b120d20fa1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76" Nov 02 17:38:23.963956 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.963928 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caee9964-de70-4471-9013-90b120d20fa1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76\" (UID: \"caee9964-de70-4471-9013-90b120d20fa1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76" Nov 02 17:38:23.992154 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.992119 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd"] Nov 02 17:38:23.999093 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:23.999057 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" Nov 02 17:38:24.016440 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:24.016413 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd"] Nov 02 17:38:24.065477 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:24.065437 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsbk5\" (UniqueName: \"kubernetes.io/projected/caee9964-de70-4471-9013-90b120d20fa1-kube-api-access-fsbk5\") pod \"payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76\" (UID: \"caee9964-de70-4471-9013-90b120d20fa1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76" Nov 02 17:38:24.065998 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:24.065496 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caee9964-de70-4471-9013-90b120d20fa1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76\" (UID: \"caee9964-de70-4471-9013-90b120d20fa1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76" Nov 02 17:38:24.066567 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:24.066543 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caee9964-de70-4471-9013-90b120d20fa1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76\" (UID: \"caee9964-de70-4471-9013-90b120d20fa1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76" Nov 02 17:38:24.094728 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:24.094692 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsbk5\" (UniqueName: \"kubernetes.io/projected/caee9964-de70-4471-9013-90b120d20fa1-kube-api-access-fsbk5\") pod \"payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76\" (UID: \"caee9964-de70-4471-9013-90b120d20fa1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76" Nov 02 17:38:24.167113 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:24.167064 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27864d8-7810-45a3-8cf1-ccc0033b4268-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd\" (UID: \"a27864d8-7810-45a3-8cf1-ccc0033b4268\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" Nov 02 17:38:24.167405 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:24.167198 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8d5z\" (UniqueName: \"kubernetes.io/projected/a27864d8-7810-45a3-8cf1-ccc0033b4268-kube-api-access-f8d5z\") pod \"payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd\" (UID: \"a27864d8-7810-45a3-8cf1-ccc0033b4268\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" Nov 02 17:38:24.201515 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:24.201476 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76" Nov 02 17:38:24.245353 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:24.245319 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl" event={"ID":"63359d54-2671-4cbc-9033-cc1716d65580","Type":"ContainerStarted","Data":"d7fdc8a53ff9c5acb20d3de679575b9f4334fbe267be6b58652ee6fecaf0cf24"} Nov 02 17:38:24.247770 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:24.247743 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm" event={"ID":"980e42a5-a602-46ea-842d-6b6914e1b870","Type":"ContainerStarted","Data":"a7b8f0a5793605ea4afdc92ff2696051117f90e0996b4e42c7a7e77e21a960de"} Nov 02 17:38:24.268706 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:24.268662 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27864d8-7810-45a3-8cf1-ccc0033b4268-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd\" (UID: \"a27864d8-7810-45a3-8cf1-ccc0033b4268\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" Nov 02 17:38:24.268909 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:24.268744 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8d5z\" (UniqueName: \"kubernetes.io/projected/a27864d8-7810-45a3-8cf1-ccc0033b4268-kube-api-access-f8d5z\") pod \"payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd\" (UID: \"a27864d8-7810-45a3-8cf1-ccc0033b4268\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" Nov 02 17:38:24.270280 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:24.270213 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27864d8-7810-45a3-8cf1-ccc0033b4268-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd\" (UID: \"a27864d8-7810-45a3-8cf1-ccc0033b4268\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" Nov 02 17:38:24.300216 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:24.300187 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8d5z\" (UniqueName: \"kubernetes.io/projected/a27864d8-7810-45a3-8cf1-ccc0033b4268-kube-api-access-f8d5z\") pod \"payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd\" (UID: \"a27864d8-7810-45a3-8cf1-ccc0033b4268\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" Nov 02 17:38:24.318109 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:24.317986 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" Nov 02 17:38:24.535700 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:24.535662 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76"] Nov 02 17:38:24.645265 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:24.645212 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd"] Nov 02 17:38:25.260354 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:25.260315 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76" event={"ID":"caee9964-de70-4471-9013-90b120d20fa1","Type":"ContainerStarted","Data":"2d3be95a5d3ce1bf9e41a790719e393a77fdfdcfcb3e311a566d7d4d0d4d670f"} Nov 02 17:38:25.260803 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:25.260383 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76" event={"ID":"caee9964-de70-4471-9013-90b120d20fa1","Type":"ContainerStarted","Data":"2fd6ef94e619bb911680715250479cb1cf51269bd04814bddd57d8ffe4044d99"} Nov 02 17:38:25.261013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:25.260952 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76" Nov 02 17:38:25.264763 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:25.264726 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" event={"ID":"75e58f61-43a2-4600-a380-549625283343","Type":"ContainerStarted","Data":"55f7c5a5ab471fc60cd7b430c1d6b6dc35a1ff33131fa7367f753f19d1b10179"} Nov 02 17:38:25.265409 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:25.265327 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" Nov 02 17:38:25.267481 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:25.267454 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" event={"ID":"bafb2593-6516-446c-b113-37bff6320729","Type":"ContainerStarted","Data":"d0ab35b8d1c48779542d7bf2aed4481f66d9eb405a3b018669c98a3f2902b337"} Nov 02 17:38:25.267923 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:25.267906 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" Nov 02 17:38:25.271203 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:25.271145 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" event={"ID":"a27864d8-7810-45a3-8cf1-ccc0033b4268","Type":"ContainerStarted","Data":"56e35e030b02e0bbeb93edf23cbd27ee008616642485520516f30425cff477b1"} Nov 02 17:38:25.271470 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:25.271436 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" event={"ID":"a27864d8-7810-45a3-8cf1-ccc0033b4268","Type":"ContainerStarted","Data":"585becf0a15ca959b002fb5d2f11b1a52461cd059f0b220482e3983083942c27"} Nov 02 17:38:25.272002 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:25.271982 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" Nov 02 17:38:25.303981 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:25.303890 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76" podStartSLOduration=1.699729891 podStartE2EDuration="2.303877303s" totalImagesPullingTime="604.147412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:25.301264345 +0000 UTC m=+523.922624122" watchObservedRunningTime="2026-11-02 17:38:25.303877303 +0000 UTC m=+523.925237079" Nov 02 17:38:25.347186 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:25.347111 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" podStartSLOduration=1.967428478 podStartE2EDuration="2.347091616s" totalImagesPullingTime="379.663138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:25.344538678 +0000 UTC m=+523.965898456" watchObservedRunningTime="2026-11-02 17:38:25.347091616 +0000 UTC m=+523.968451397" Nov 02 17:38:25.389671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:25.389606 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" podStartSLOduration=1.243790328 podStartE2EDuration="4.389587303s" totalImagesPullingTime="3.145796975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:25.385149958 +0000 UTC m=+524.006509736" watchObservedRunningTime="2026-11-02 17:38:25.389587303 +0000 UTC m=+524.010947078" Nov 02 17:38:25.429303 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:25.429243 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" podStartSLOduration=1.270218734 podStartE2EDuration="4.429224729s" totalImagesPullingTime="3.159005995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:25.428247703 +0000 UTC m=+524.049607491" watchObservedRunningTime="2026-11-02 17:38:25.429224729 +0000 UTC m=+524.050584507" Nov 02 17:38:26.278719 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.278686 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl" event={"ID":"63359d54-2671-4cbc-9033-cc1716d65580","Type":"ContainerStarted","Data":"4b40d12ad66cfa3569eb26aba53a7609344938b410d508b0a2e0afcdc5c8e16f"} Nov 02 17:38:26.279177 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.278747 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl" Nov 02 17:38:26.280337 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.280312 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm" event={"ID":"980e42a5-a602-46ea-842d-6b6914e1b870","Type":"ContainerStarted","Data":"23651af746e5a53fcd157d294c2213c9804a085898f5cde03154f455fe51000b"} Nov 02 17:38:26.280525 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.280501 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm" Nov 02 17:38:26.323820 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.323767 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl" podStartSLOduration=1.504219196 podStartE2EDuration="3.323750934s" totalImagesPullingTime="1.819531738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:26.320614689 +0000 UTC m=+524.941974464" watchObservedRunningTime="2026-11-02 17:38:26.323750934 +0000 UTC m=+524.945110708" Nov 02 17:38:26.365583 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.365555 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t"] Nov 02 17:38:26.367371 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.367332 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm" podStartSLOduration=1.6953398210000001 podStartE2EDuration="3.367319812s" totalImagesPullingTime="1.671979991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:26.364259597 +0000 UTC m=+524.985619373" watchObservedRunningTime="2026-11-02 17:38:26.367319812 +0000 UTC m=+524.988679627" Nov 02 17:38:26.369501 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.369486 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" Nov 02 17:38:26.385820 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.385797 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-863b87\"" Nov 02 17:38:26.385986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.385832 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-863b87-dockercfg-ch85x\"" Nov 02 17:38:26.389497 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.389470 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t"] Nov 02 17:38:26.457916 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.454579 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr"] Nov 02 17:38:26.462203 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.462180 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" Nov 02 17:38:26.477715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.477686 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr"] Nov 02 17:38:26.493829 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.493797 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/840ac652-3ceb-43ef-91b3-42c19179fe01-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t\" (UID: \"840ac652-3ceb-43ef-91b3-42c19179fe01\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" Nov 02 17:38:26.494020 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.493962 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2mwk\" (UniqueName: \"kubernetes.io/projected/840ac652-3ceb-43ef-91b3-42c19179fe01-kube-api-access-s2mwk\") pod \"payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t\" (UID: \"840ac652-3ceb-43ef-91b3-42c19179fe01\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" Nov 02 17:38:26.594986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.594946 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2mwk\" (UniqueName: \"kubernetes.io/projected/840ac652-3ceb-43ef-91b3-42c19179fe01-kube-api-access-s2mwk\") pod \"payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t\" (UID: \"840ac652-3ceb-43ef-91b3-42c19179fe01\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" Nov 02 17:38:26.594986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.594991 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/840ac652-3ceb-43ef-91b3-42c19179fe01-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t\" (UID: \"840ac652-3ceb-43ef-91b3-42c19179fe01\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" Nov 02 17:38:26.595193 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.595032 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d179ee-224e-4375-8cf1-863a774e1cb6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr\" (UID: \"79d179ee-224e-4375-8cf1-863a774e1cb6\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" Nov 02 17:38:26.595231 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.595204 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnw6b\" (UniqueName: \"kubernetes.io/projected/79d179ee-224e-4375-8cf1-863a774e1cb6-kube-api-access-dnw6b\") pod \"payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr\" (UID: \"79d179ee-224e-4375-8cf1-863a774e1cb6\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" Nov 02 17:38:26.595757 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.595738 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/840ac652-3ceb-43ef-91b3-42c19179fe01-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t\" (UID: \"840ac652-3ceb-43ef-91b3-42c19179fe01\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" Nov 02 17:38:26.618841 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.618805 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2mwk\" (UniqueName: \"kubernetes.io/projected/840ac652-3ceb-43ef-91b3-42c19179fe01-kube-api-access-s2mwk\") pod \"payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t\" (UID: \"840ac652-3ceb-43ef-91b3-42c19179fe01\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" Nov 02 17:38:26.683360 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.683327 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" Nov 02 17:38:26.696004 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.695974 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnw6b\" (UniqueName: \"kubernetes.io/projected/79d179ee-224e-4375-8cf1-863a774e1cb6-kube-api-access-dnw6b\") pod \"payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr\" (UID: \"79d179ee-224e-4375-8cf1-863a774e1cb6\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" Nov 02 17:38:26.696140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.696077 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d179ee-224e-4375-8cf1-863a774e1cb6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr\" (UID: \"79d179ee-224e-4375-8cf1-863a774e1cb6\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" Nov 02 17:38:26.696777 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.696700 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d179ee-224e-4375-8cf1-863a774e1cb6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr\" (UID: \"79d179ee-224e-4375-8cf1-863a774e1cb6\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" Nov 02 17:38:26.719874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.719830 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnw6b\" (UniqueName: \"kubernetes.io/projected/79d179ee-224e-4375-8cf1-863a774e1cb6-kube-api-access-dnw6b\") pod \"payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr\" (UID: \"79d179ee-224e-4375-8cf1-863a774e1cb6\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" Nov 02 17:38:26.777358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.777326 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" Nov 02 17:38:26.998483 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:38:26.998449 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod840ac652_3ceb_43ef_91b3_42c19179fe01.slice/crio-27b66f7dbec561f481457e5d5827bba170838266133e50d796ab9bb7176af262 WatchSource:0}: Error finding container 27b66f7dbec561f481457e5d5827bba170838266133e50d796ab9bb7176af262: Status 404 returned error can't find the container with id 27b66f7dbec561f481457e5d5827bba170838266133e50d796ab9bb7176af262 Nov 02 17:38:26.999289 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:26.999264 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t"] Nov 02 17:38:27.080389 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:38:27.080362 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d179ee_224e_4375_8cf1_863a774e1cb6.slice/crio-6f4461a7a8ce32fb9a58a522fa49cb2d28817d366d80f6afcac7d659b2803b21 WatchSource:0}: Error finding container 6f4461a7a8ce32fb9a58a522fa49cb2d28817d366d80f6afcac7d659b2803b21: Status 404 returned error can't find the container with id 6f4461a7a8ce32fb9a58a522fa49cb2d28817d366d80f6afcac7d659b2803b21 Nov 02 17:38:27.080835 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:27.080807 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr"] Nov 02 17:38:27.287753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:27.287655 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" event={"ID":"79d179ee-224e-4375-8cf1-863a774e1cb6","Type":"ContainerStarted","Data":"5f03e926258629d0b4e3feec69fbaeb084795547f69ae4c0da0fd9344149fae6"} Nov 02 17:38:27.287753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:27.287720 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" Nov 02 17:38:27.287753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:27.287746 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" event={"ID":"79d179ee-224e-4375-8cf1-863a774e1cb6","Type":"ContainerStarted","Data":"6f4461a7a8ce32fb9a58a522fa49cb2d28817d366d80f6afcac7d659b2803b21"} Nov 02 17:38:27.289543 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:27.289511 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" event={"ID":"840ac652-3ceb-43ef-91b3-42c19179fe01","Type":"ContainerStarted","Data":"2b63690d74f4c643dd8a309039390ef233f26f0965fc14d8608ea3a86235f084"} Nov 02 17:38:27.289711 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:27.289597 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" event={"ID":"840ac652-3ceb-43ef-91b3-42c19179fe01","Type":"ContainerStarted","Data":"27b66f7dbec561f481457e5d5827bba170838266133e50d796ab9bb7176af262"} Nov 02 17:38:27.290125 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:27.290103 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" Nov 02 17:38:27.347135 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:27.347082 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" podStartSLOduration=1.3470663379999999 podStartE2EDuration="1.347066338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:27.344339703 +0000 UTC m=+525.965699479" watchObservedRunningTime="2026-11-02 17:38:27.347066338 +0000 UTC m=+525.968426113" Nov 02 17:38:27.388055 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:27.388003 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" podStartSLOduration=1.387989552 podStartE2EDuration="1.387989552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:27.385077477 +0000 UTC m=+526.006437252" watchObservedRunningTime="2026-11-02 17:38:27.387989552 +0000 UTC m=+526.009349327" Nov 02 17:38:28.964279 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:28.964215 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2"] Nov 02 17:38:28.968196 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:28.968171 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" Nov 02 17:38:28.995437 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:28.995408 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2"] Nov 02 17:38:28.998830 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:28.998808 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-069fa5\"" Nov 02 17:38:29.041183 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.041145 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-069fa5-dockercfg-jzgfw\"" Nov 02 17:38:29.044909 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.044880 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp"] Nov 02 17:38:29.048648 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.048629 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" Nov 02 17:38:29.070338 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.070310 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp"] Nov 02 17:38:29.119330 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.119288 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c40d2ec9-ed70-417a-8c81-2a93ae477425-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2\" (UID: \"c40d2ec9-ed70-417a-8c81-2a93ae477425\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" Nov 02 17:38:29.119498 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.119341 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jffb6\" (UniqueName: \"kubernetes.io/projected/c40d2ec9-ed70-417a-8c81-2a93ae477425-kube-api-access-jffb6\") pod \"payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2\" (UID: \"c40d2ec9-ed70-417a-8c81-2a93ae477425\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" Nov 02 17:38:29.220898 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.220788 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51de62e-89f5-4ac1-9f79-b64defc7e887-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp\" (UID: \"b51de62e-89f5-4ac1-9f79-b64defc7e887\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" Nov 02 17:38:29.220898 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.220879 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c40d2ec9-ed70-417a-8c81-2a93ae477425-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2\" (UID: \"c40d2ec9-ed70-417a-8c81-2a93ae477425\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" Nov 02 17:38:29.221067 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.220945 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jffb6\" (UniqueName: \"kubernetes.io/projected/c40d2ec9-ed70-417a-8c81-2a93ae477425-kube-api-access-jffb6\") pod \"payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2\" (UID: \"c40d2ec9-ed70-417a-8c81-2a93ae477425\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" Nov 02 17:38:29.221123 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.221072 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brnvz\" (UniqueName: \"kubernetes.io/projected/b51de62e-89f5-4ac1-9f79-b64defc7e887-kube-api-access-brnvz\") pod \"payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp\" (UID: \"b51de62e-89f5-4ac1-9f79-b64defc7e887\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" Nov 02 17:38:29.221538 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.221522 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c40d2ec9-ed70-417a-8c81-2a93ae477425-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2\" (UID: \"c40d2ec9-ed70-417a-8c81-2a93ae477425\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" Nov 02 17:38:29.244899 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.244869 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jffb6\" (UniqueName: \"kubernetes.io/projected/c40d2ec9-ed70-417a-8c81-2a93ae477425-kube-api-access-jffb6\") pod \"payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2\" (UID: \"c40d2ec9-ed70-417a-8c81-2a93ae477425\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" Nov 02 17:38:29.282355 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.282328 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" Nov 02 17:38:29.322610 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.322567 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brnvz\" (UniqueName: \"kubernetes.io/projected/b51de62e-89f5-4ac1-9f79-b64defc7e887-kube-api-access-brnvz\") pod \"payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp\" (UID: \"b51de62e-89f5-4ac1-9f79-b64defc7e887\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" Nov 02 17:38:29.322790 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.322673 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51de62e-89f5-4ac1-9f79-b64defc7e887-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp\" (UID: \"b51de62e-89f5-4ac1-9f79-b64defc7e887\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" Nov 02 17:38:29.323387 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.323363 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51de62e-89f5-4ac1-9f79-b64defc7e887-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp\" (UID: \"b51de62e-89f5-4ac1-9f79-b64defc7e887\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" Nov 02 17:38:29.352421 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.352394 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brnvz\" (UniqueName: \"kubernetes.io/projected/b51de62e-89f5-4ac1-9f79-b64defc7e887-kube-api-access-brnvz\") pod \"payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp\" (UID: \"b51de62e-89f5-4ac1-9f79-b64defc7e887\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" Nov 02 17:38:29.363218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.363193 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" Nov 02 17:38:29.602879 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.602826 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2"] Nov 02 17:38:29.603468 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:38:29.603436 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40d2ec9_ed70_417a_8c81_2a93ae477425.slice/crio-38f9f7dfd2e32792d5d51863d14b0ce6d6eb61d6e1c98a6b4276ce25814b3a70 WatchSource:0}: Error finding container 38f9f7dfd2e32792d5d51863d14b0ce6d6eb61d6e1c98a6b4276ce25814b3a70: Status 404 returned error can't find the container with id 38f9f7dfd2e32792d5d51863d14b0ce6d6eb61d6e1c98a6b4276ce25814b3a70 Nov 02 17:38:29.671789 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:38:29.671760 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51de62e_89f5_4ac1_9f79_b64defc7e887.slice/crio-ef533dcbe4a136181501e3bab404e5c442130923dcccf0da172724aedd88d92e WatchSource:0}: Error finding container ef533dcbe4a136181501e3bab404e5c442130923dcccf0da172724aedd88d92e: Status 404 returned error can't find the container with id ef533dcbe4a136181501e3bab404e5c442130923dcccf0da172724aedd88d92e Nov 02 17:38:29.673167 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:29.673102 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp"] Nov 02 17:38:30.311451 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:30.311411 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" event={"ID":"b51de62e-89f5-4ac1-9f79-b64defc7e887","Type":"ContainerStarted","Data":"bc3c8d206613a2c3a47429339ebf144a63d7c862f515967e8a9996b6f81fc386"} Nov 02 17:38:30.311965 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:30.311468 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" event={"ID":"b51de62e-89f5-4ac1-9f79-b64defc7e887","Type":"ContainerStarted","Data":"ef533dcbe4a136181501e3bab404e5c442130923dcccf0da172724aedd88d92e"} Nov 02 17:38:30.311965 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:30.311620 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" Nov 02 17:38:30.313000 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:30.312971 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" event={"ID":"c40d2ec9-ed70-417a-8c81-2a93ae477425","Type":"ContainerStarted","Data":"422e6754eac6b47f2673e24062270d5b94a01b65f3ba7ebfd1ccdc50c06420c5"} Nov 02 17:38:30.313099 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:30.313021 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" Nov 02 17:38:30.313099 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:30.313040 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" event={"ID":"c40d2ec9-ed70-417a-8c81-2a93ae477425","Type":"ContainerStarted","Data":"38f9f7dfd2e32792d5d51863d14b0ce6d6eb61d6e1c98a6b4276ce25814b3a70"} Nov 02 17:38:30.354620 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:30.354568 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" podStartSLOduration=1.35455462 podStartE2EDuration="1.35455462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:30.35230988 +0000 UTC m=+528.973669653" watchObservedRunningTime="2026-11-02 17:38:30.35455462 +0000 UTC m=+528.975914396" Nov 02 17:38:30.394977 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:30.394926 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" podStartSLOduration=2.394912615 podStartE2EDuration="2.394912615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:30.393130504 +0000 UTC m=+529.014490281" watchObservedRunningTime="2026-11-02 17:38:30.394912615 +0000 UTC m=+529.016272390" Nov 02 17:38:32.796464 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:32.796428 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4"] Nov 02 17:38:32.800631 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:32.800609 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" Nov 02 17:38:32.832183 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:32.832152 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 17:38:32.832336 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:32.832287 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-863b87-serving-cert\"" Nov 02 17:38:32.832397 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:32.832385 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gml5d\"" Nov 02 17:38:32.837315 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:32.837288 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4"] Nov 02 17:38:32.963905 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:32.963870 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0839624-4c7c-42fe-acc1-7515cea20996-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-863b87-5886444974-qm9z4\" (UID: \"e0839624-4c7c-42fe-acc1-7515cea20996\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" Nov 02 17:38:32.964066 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:32.963920 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0839624-4c7c-42fe-acc1-7515cea20996-maas-api-tls\") pod \"maas-api-e2e-worker-w5-863b87-5886444974-qm9z4\" (UID: \"e0839624-4c7c-42fe-acc1-7515cea20996\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" Nov 02 17:38:32.964066 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:32.964000 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pbtp\" (UniqueName: \"kubernetes.io/projected/e0839624-4c7c-42fe-acc1-7515cea20996-kube-api-access-2pbtp\") pod \"maas-api-e2e-worker-w5-863b87-5886444974-qm9z4\" (UID: \"e0839624-4c7c-42fe-acc1-7515cea20996\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" Nov 02 17:38:32.964162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:32.964066 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0839624-4c7c-42fe-acc1-7515cea20996-metrics-certs\") pod \"maas-api-e2e-worker-w5-863b87-5886444974-qm9z4\" (UID: \"e0839624-4c7c-42fe-acc1-7515cea20996\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" Nov 02 17:38:33.065225 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:33.065186 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0839624-4c7c-42fe-acc1-7515cea20996-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-863b87-5886444974-qm9z4\" (UID: \"e0839624-4c7c-42fe-acc1-7515cea20996\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" Nov 02 17:38:33.065225 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:33.065241 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0839624-4c7c-42fe-acc1-7515cea20996-maas-api-tls\") pod \"maas-api-e2e-worker-w5-863b87-5886444974-qm9z4\" (UID: \"e0839624-4c7c-42fe-acc1-7515cea20996\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" Nov 02 17:38:33.065541 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:33.065268 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pbtp\" (UniqueName: \"kubernetes.io/projected/e0839624-4c7c-42fe-acc1-7515cea20996-kube-api-access-2pbtp\") pod \"maas-api-e2e-worker-w5-863b87-5886444974-qm9z4\" (UID: \"e0839624-4c7c-42fe-acc1-7515cea20996\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" Nov 02 17:38:33.065880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:33.065816 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0839624-4c7c-42fe-acc1-7515cea20996-metrics-certs\") pod \"maas-api-e2e-worker-w5-863b87-5886444974-qm9z4\" (UID: \"e0839624-4c7c-42fe-acc1-7515cea20996\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" Nov 02 17:38:33.067171 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:33.067141 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0839624-4c7c-42fe-acc1-7515cea20996-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-863b87-5886444974-qm9z4\" (UID: \"e0839624-4c7c-42fe-acc1-7515cea20996\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" Nov 02 17:38:33.068800 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:33.068772 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0839624-4c7c-42fe-acc1-7515cea20996-maas-api-tls\") pod \"maas-api-e2e-worker-w5-863b87-5886444974-qm9z4\" (UID: \"e0839624-4c7c-42fe-acc1-7515cea20996\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" Nov 02 17:38:33.069118 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:33.069089 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0839624-4c7c-42fe-acc1-7515cea20996-metrics-certs\") pod \"maas-api-e2e-worker-w5-863b87-5886444974-qm9z4\" (UID: \"e0839624-4c7c-42fe-acc1-7515cea20996\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" Nov 02 17:38:33.089247 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:33.089217 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pbtp\" (UniqueName: \"kubernetes.io/projected/e0839624-4c7c-42fe-acc1-7515cea20996-kube-api-access-2pbtp\") pod \"maas-api-e2e-worker-w5-863b87-5886444974-qm9z4\" (UID: \"e0839624-4c7c-42fe-acc1-7515cea20996\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" Nov 02 17:38:33.115074 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:33.115036 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" Nov 02 17:38:33.440885 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:33.440827 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4"] Nov 02 17:38:33.442195 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:38:33.442169 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0839624_4c7c_42fe_acc1_7515cea20996.slice/crio-d3562ae869dc6c9d565bc7176ef4282935d047c482c3ffefa30fdfaec1629c75 WatchSource:0}: Error finding container d3562ae869dc6c9d565bc7176ef4282935d047c482c3ffefa30fdfaec1629c75: Status 404 returned error can't find the container with id d3562ae869dc6c9d565bc7176ef4282935d047c482c3ffefa30fdfaec1629c75 Nov 02 17:38:34.342045 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:34.342000 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" event={"ID":"e0839624-4c7c-42fe-acc1-7515cea20996","Type":"ContainerStarted","Data":"d3562ae869dc6c9d565bc7176ef4282935d047c482c3ffefa30fdfaec1629c75"} Nov 02 17:38:35.349790 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:35.349752 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" event={"ID":"e0839624-4c7c-42fe-acc1-7515cea20996","Type":"ContainerStarted","Data":"771bacacc944ca343be59572a5feae7f535fc0d411f9fb6075da32179e97ca61"} Nov 02 17:38:35.350192 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:35.349907 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" Nov 02 17:38:35.402768 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:35.402718 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" podStartSLOduration=1.918680213 podStartE2EDuration="3.402705453s" totalImagesPullingTime="1.48402524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:35.401154085 +0000 UTC m=+534.022513860" watchObservedRunningTime="2026-11-02 17:38:35.402705453 +0000 UTC m=+534.024065229" Nov 02 17:38:37.293292 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:37.293259 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm" Nov 02 17:38:37.293748 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:37.293483 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" Nov 02 17:38:37.293748 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:37.293554 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" Nov 02 17:38:37.293844 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:37.293796 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76" Nov 02 17:38:37.293844 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:37.293860 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" Nov 02 17:38:37.293984 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:37.293962 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl" Nov 02 17:38:38.299989 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:38.299958 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" Nov 02 17:38:39.307723 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:39.307695 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" Nov 02 17:38:41.323335 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:41.323302 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" Nov 02 17:38:41.323751 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:41.323383 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" Nov 02 17:38:41.362190 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:41.362163 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" Nov 02 17:38:51.772714 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:38:51.772680 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:38:53.403792 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.403744 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d"] Nov 02 17:38:53.411339 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.411304 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:53.431515 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.431480 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d"] Nov 02 17:38:53.441380 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.441351 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7df526\"/\"openshift-service-ca.crt\"" Nov 02 17:38:53.441560 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.441387 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7df526\"/\"kube-root-ca.crt\"" Nov 02 17:38:53.441642 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.441428 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7df526\"/\"simulator-w6-7df526-kserve-self-signed-certs\"" Nov 02 17:38:53.441642 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.441488 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7df526\"/\"default-dockercfg-gs46w\"" Nov 02 17:38:53.459478 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.459448 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-tmp-dir\") pod \"simulator-w6-7df526-kserve-76c8fc647f-qx96d\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:53.459478 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.459482 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-dshm\") pod \"simulator-w6-7df526-kserve-76c8fc647f-qx96d\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:53.459711 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.459507 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-tls-certs\") pod \"simulator-w6-7df526-kserve-76c8fc647f-qx96d\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:53.459711 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.459564 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-model-cache\") pod \"simulator-w6-7df526-kserve-76c8fc647f-qx96d\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:53.459711 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.459683 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5xgb\" (UniqueName: \"kubernetes.io/projected/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-kube-api-access-b5xgb\") pod \"simulator-w6-7df526-kserve-76c8fc647f-qx96d\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:53.459869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.459724 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-home\") pod \"simulator-w6-7df526-kserve-76c8fc647f-qx96d\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:53.560394 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.560358 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-tmp-dir\") pod \"simulator-w6-7df526-kserve-76c8fc647f-qx96d\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:53.560583 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.560402 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-dshm\") pod \"simulator-w6-7df526-kserve-76c8fc647f-qx96d\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:53.560583 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.560453 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-tls-certs\") pod \"simulator-w6-7df526-kserve-76c8fc647f-qx96d\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:53.560583 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.560484 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-model-cache\") pod \"simulator-w6-7df526-kserve-76c8fc647f-qx96d\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:53.560583 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.560546 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5xgb\" (UniqueName: \"kubernetes.io/projected/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-kube-api-access-b5xgb\") pod \"simulator-w6-7df526-kserve-76c8fc647f-qx96d\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:53.560795 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.560639 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-home\") pod \"simulator-w6-7df526-kserve-76c8fc647f-qx96d\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:53.560871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.560805 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-tmp-dir\") pod \"simulator-w6-7df526-kserve-76c8fc647f-qx96d\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:53.561153 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.561128 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-model-cache\") pod \"simulator-w6-7df526-kserve-76c8fc647f-qx96d\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:53.561338 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.561318 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-home\") pod \"simulator-w6-7df526-kserve-76c8fc647f-qx96d\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:53.562817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.562790 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-dshm\") pod \"simulator-w6-7df526-kserve-76c8fc647f-qx96d\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:53.563175 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.563158 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-tls-certs\") pod \"simulator-w6-7df526-kserve-76c8fc647f-qx96d\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:53.584536 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.584515 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5xgb\" (UniqueName: \"kubernetes.io/projected/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-kube-api-access-b5xgb\") pod \"simulator-w6-7df526-kserve-76c8fc647f-qx96d\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:53.725227 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:53.725135 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:38:54.032369 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:54.032325 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d"] Nov 02 17:38:54.034953 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:38:54.034914 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebd4fcc7_38eb_40ff_b389_8b1fce29d132.slice/crio-86dfaf21f5945982124aeb1617dc2ab93ce7d6f2e997aaf09ee38982243bfaf6 WatchSource:0}: Error finding container 86dfaf21f5945982124aeb1617dc2ab93ce7d6f2e997aaf09ee38982243bfaf6: Status 404 returned error can't find the container with id 86dfaf21f5945982124aeb1617dc2ab93ce7d6f2e997aaf09ee38982243bfaf6 Nov 02 17:38:54.489043 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:54.488979 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" event={"ID":"ebd4fcc7-38eb-40ff-b389-8b1fce29d132","Type":"ContainerStarted","Data":"86dfaf21f5945982124aeb1617dc2ab93ce7d6f2e997aaf09ee38982243bfaf6"} Nov 02 17:38:55.499222 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:55.499145 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" event={"ID":"ebd4fcc7-38eb-40ff-b389-8b1fce29d132","Type":"ContainerStarted","Data":"e5d7b58377638c9c2714dfa776c4a2700db884b20e1de81751eb49b3f52d9bfd"} Nov 02 17:38:55.546823 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:55.546756 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podStartSLOduration=1.406868265 podStartE2EDuration="2.546734987s" totalImagesPullingTime="1.139866722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:55.545103978 +0000 UTC m=+554.166463755" watchObservedRunningTime="2026-11-02 17:38:55.546734987 +0000 UTC m=+554.168094767" Nov 02 17:38:55.725376 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:38:55.725342 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:39:03.725308 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:03.725264 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:39:03.738049 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:03.738012 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:39:04.389075 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.389044 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq"] Nov 02 17:39:04.395363 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.395339 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:04.415383 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.415353 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq"] Nov 02 17:39:04.422940 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.422913 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2899ab\"/\"openshift-service-ca.crt\"" Nov 02 17:39:04.423128 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.423108 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2899ab\"/\"kube-root-ca.crt\"" Nov 02 17:39:04.423386 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.423358 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2899ab\"/\"default-dockercfg-zlvc8\"" Nov 02 17:39:04.423573 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.423552 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2899ab\"/\"simulator-w4-2899ab-kserve-self-signed-certs\"" Nov 02 17:39:04.468938 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.468913 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce731d39-0cbe-4322-aa80-1d2606c3970a-tls-certs\") pod \"simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:04.469077 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.468942 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxn5k\" (UniqueName: \"kubernetes.io/projected/ce731d39-0cbe-4322-aa80-1d2606c3970a-kube-api-access-xxn5k\") pod \"simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:04.469077 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.468984 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-dshm\") pod \"simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:04.469077 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.469038 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-home\") pod \"simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:04.469189 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.469147 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-tmp-dir\") pod \"simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:04.469189 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.469177 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-model-cache\") pod \"simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:04.570148 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.570115 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-dshm\") pod \"simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:04.570404 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.570376 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-home\") pod \"simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:04.570587 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.570569 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-tmp-dir\") pod \"simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:04.570670 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.570622 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-model-cache\") pod \"simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:04.570768 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.570742 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-home\") pod \"simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:04.571552 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.570738 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce731d39-0cbe-4322-aa80-1d2606c3970a-tls-certs\") pod \"simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:04.571781 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.571753 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxn5k\" (UniqueName: \"kubernetes.io/projected/ce731d39-0cbe-4322-aa80-1d2606c3970a-kube-api-access-xxn5k\") pod \"simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:04.572054 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.570940 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-tmp-dir\") pod \"simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:04.572298 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.571499 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-model-cache\") pod \"simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:04.573025 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.572994 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-dshm\") pod \"simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:04.573434 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.573412 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce731d39-0cbe-4322-aa80-1d2606c3970a-tls-certs\") pod \"simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:04.574741 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.574721 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:39:04.598671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.598642 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxn5k\" (UniqueName: \"kubernetes.io/projected/ce731d39-0cbe-4322-aa80-1d2606c3970a-kube-api-access-xxn5k\") pod \"simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:04.709470 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:04.709393 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:05.028036 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:05.028010 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq"] Nov 02 17:39:05.028955 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:39:05.028930 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce731d39_0cbe_4322_aa80_1d2606c3970a.slice/crio-af2a38f7e8c604398076bc64c42dd9600e8d2e7b02eb8f19c04e8ec7366f4a8c WatchSource:0}: Error finding container af2a38f7e8c604398076bc64c42dd9600e8d2e7b02eb8f19c04e8ec7366f4a8c: Status 404 returned error can't find the container with id af2a38f7e8c604398076bc64c42dd9600e8d2e7b02eb8f19c04e8ec7366f4a8c Nov 02 17:39:05.574106 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:05.574061 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" event={"ID":"ce731d39-0cbe-4322-aa80-1d2606c3970a","Type":"ContainerStarted","Data":"a34c27d1d85194b32548c9b34ab349e9dfa062f7b1e69dded2140a551d123bf0"} Nov 02 17:39:05.574283 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:05.574111 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" event={"ID":"ce731d39-0cbe-4322-aa80-1d2606c3970a","Type":"ContainerStarted","Data":"af2a38f7e8c604398076bc64c42dd9600e8d2e7b02eb8f19c04e8ec7366f4a8c"} Nov 02 17:39:05.619640 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:05.619586 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podStartSLOduration=1.6195735249999998 podStartE2EDuration="1.619573525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:05.615933477 +0000 UTC m=+564.237293254" watchObservedRunningTime="2026-11-02 17:39:05.619573525 +0000 UTC m=+564.240933300" Nov 02 17:39:05.709689 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:05.709655 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:14.710234 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:14.710197 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:14.723215 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:14.723181 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:15.648711 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.648670 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:39:15.681505 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.681476 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw"] Nov 02 17:39:15.685657 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.685641 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:15.700111 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.700089 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw"] Nov 02 17:39:15.708400 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.708377 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"default-dockercfg-8dv9q\"" Nov 02 17:39:15.708915 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.708897 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"simulator-w0-069fa5-kserve-self-signed-certs\"" Nov 02 17:39:15.750741 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.750710 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"openshift-service-ca.crt\"" Nov 02 17:39:15.751148 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.750743 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"kube-root-ca.crt\"" Nov 02 17:39:15.786926 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.786893 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8jlh\" (UniqueName: \"kubernetes.io/projected/b52df912-8bb6-4b46-b2bc-c51da240b140-kube-api-access-w8jlh\") pod \"simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:15.787111 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.786938 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-dshm\") pod \"simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:15.787111 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.787030 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-tmp-dir\") pod \"simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:15.787111 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.787073 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b52df912-8bb6-4b46-b2bc-c51da240b140-tls-certs\") pod \"simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:15.787304 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.787279 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-model-cache\") pod \"simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:15.787425 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.787336 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-home\") pod \"simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:15.888969 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.888932 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-model-cache\") pod \"simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:15.889148 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.888984 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-home\") pod \"simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:15.889148 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.889033 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8jlh\" (UniqueName: \"kubernetes.io/projected/b52df912-8bb6-4b46-b2bc-c51da240b140-kube-api-access-w8jlh\") pod \"simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:15.889148 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.889052 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-dshm\") pod \"simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:15.889329 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.889164 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-tmp-dir\") pod \"simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:15.889329 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.889277 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b52df912-8bb6-4b46-b2bc-c51da240b140-tls-certs\") pod \"simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:15.889458 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.889437 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-home\") pod \"simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:15.889507 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.889487 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-model-cache\") pod \"simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:15.889548 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.889518 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-tmp-dir\") pod \"simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:15.891990 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.891958 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-dshm\") pod \"simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:15.891990 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.891971 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b52df912-8bb6-4b46-b2bc-c51da240b140-tls-certs\") pod \"simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:15.914384 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.914291 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8jlh\" (UniqueName: \"kubernetes.io/projected/b52df912-8bb6-4b46-b2bc-c51da240b140-kube-api-access-w8jlh\") pod \"simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:15.999407 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:15.999374 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:16.324039 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:16.324000 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw"] Nov 02 17:39:16.344983 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:39:16.344949 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb52df912_8bb6_4b46_b2bc_c51da240b140.slice/crio-9827f5a6b4dd6c73db8306d6af8ce58916ca6b773870d0e4c6a71694fd8ff124 WatchSource:0}: Error finding container 9827f5a6b4dd6c73db8306d6af8ce58916ca6b773870d0e4c6a71694fd8ff124: Status 404 returned error can't find the container with id 9827f5a6b4dd6c73db8306d6af8ce58916ca6b773870d0e4c6a71694fd8ff124 Nov 02 17:39:16.646020 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:16.645922 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" event={"ID":"b52df912-8bb6-4b46-b2bc-c51da240b140","Type":"ContainerStarted","Data":"1dfcd2cecffe65cbd9feadf43f217de3c10e8edb461d22eeb99da7f151fe260e"} Nov 02 17:39:16.646020 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:16.645971 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" event={"ID":"b52df912-8bb6-4b46-b2bc-c51da240b140","Type":"ContainerStarted","Data":"9827f5a6b4dd6c73db8306d6af8ce58916ca6b773870d0e4c6a71694fd8ff124"} Nov 02 17:39:16.690078 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:16.690026 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podStartSLOduration=1.6900135120000002 podStartE2EDuration="1.690013512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:16.688256322 +0000 UTC m=+575.309616098" watchObservedRunningTime="2026-11-02 17:39:16.690013512 +0000 UTC m=+575.311373288" Nov 02 17:39:17.000425 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:17.000352 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:21.771589 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:39:21.771551 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:39:26.000184 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:26.000144 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:26.013523 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:26.013493 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:26.721781 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:26.721748 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:39:37.000526 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.000491 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf"] Nov 02 17:39:37.005251 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.005220 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.023782 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.023750 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf"] Nov 02 17:39:37.031214 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.031188 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2899ab\"/\"premium-w4-2899ab-kserve-self-signed-certs\"" Nov 02 17:39:37.089589 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.089557 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxhq5\" (UniqueName: \"kubernetes.io/projected/e50a1a79-1669-4ad8-89bb-d462a0e80100-kube-api-access-wxhq5\") pod \"premium-w4-2899ab-kserve-5d6f649c8c-mmwpf\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.089589 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.089592 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-home\") pod \"premium-w4-2899ab-kserve-5d6f649c8c-mmwpf\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.089879 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.089640 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-tmp-dir\") pod \"premium-w4-2899ab-kserve-5d6f649c8c-mmwpf\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.089879 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.089800 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e50a1a79-1669-4ad8-89bb-d462a0e80100-tls-certs\") pod \"premium-w4-2899ab-kserve-5d6f649c8c-mmwpf\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.090015 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.089878 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-dshm\") pod \"premium-w4-2899ab-kserve-5d6f649c8c-mmwpf\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.090015 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.089941 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-model-cache\") pod \"premium-w4-2899ab-kserve-5d6f649c8c-mmwpf\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.191301 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.191257 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-dshm\") pod \"premium-w4-2899ab-kserve-5d6f649c8c-mmwpf\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.191301 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.191308 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-model-cache\") pod \"premium-w4-2899ab-kserve-5d6f649c8c-mmwpf\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.191593 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.191387 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxhq5\" (UniqueName: \"kubernetes.io/projected/e50a1a79-1669-4ad8-89bb-d462a0e80100-kube-api-access-wxhq5\") pod \"premium-w4-2899ab-kserve-5d6f649c8c-mmwpf\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.191593 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.191416 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-home\") pod \"premium-w4-2899ab-kserve-5d6f649c8c-mmwpf\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.191593 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.191492 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-tmp-dir\") pod \"premium-w4-2899ab-kserve-5d6f649c8c-mmwpf\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.191593 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.191567 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e50a1a79-1669-4ad8-89bb-d462a0e80100-tls-certs\") pod \"premium-w4-2899ab-kserve-5d6f649c8c-mmwpf\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.192234 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.192140 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-home\") pod \"premium-w4-2899ab-kserve-5d6f649c8c-mmwpf\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.192234 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.192154 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-model-cache\") pod \"premium-w4-2899ab-kserve-5d6f649c8c-mmwpf\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.192449 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.192258 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-tmp-dir\") pod \"premium-w4-2899ab-kserve-5d6f649c8c-mmwpf\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.195673 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.194682 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-dshm\") pod \"premium-w4-2899ab-kserve-5d6f649c8c-mmwpf\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.195673 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.195001 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e50a1a79-1669-4ad8-89bb-d462a0e80100-tls-certs\") pod \"premium-w4-2899ab-kserve-5d6f649c8c-mmwpf\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.215526 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.215490 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxhq5\" (UniqueName: \"kubernetes.io/projected/e50a1a79-1669-4ad8-89bb-d462a0e80100-kube-api-access-wxhq5\") pod \"premium-w4-2899ab-kserve-5d6f649c8c-mmwpf\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.323654 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.323625 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:37.627357 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.627323 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf"] Nov 02 17:39:37.628808 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:39:37.628777 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode50a1a79_1669_4ad8_89bb_d462a0e80100.slice/crio-32faafefc61dac647c3b01b2e09719625ec7c08b9516b9ff6668cd8b14dbac93 WatchSource:0}: Error finding container 32faafefc61dac647c3b01b2e09719625ec7c08b9516b9ff6668cd8b14dbac93: Status 404 returned error can't find the container with id 32faafefc61dac647c3b01b2e09719625ec7c08b9516b9ff6668cd8b14dbac93 Nov 02 17:39:37.789438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.789404 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" event={"ID":"e50a1a79-1669-4ad8-89bb-d462a0e80100","Type":"ContainerStarted","Data":"27d395bb387cc590929f34fa7a11e8adcd0345d8ec44c3d3c2a7a5410fd77099"} Nov 02 17:39:37.789438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.789442 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" event={"ID":"e50a1a79-1669-4ad8-89bb-d462a0e80100","Type":"ContainerStarted","Data":"32faafefc61dac647c3b01b2e09719625ec7c08b9516b9ff6668cd8b14dbac93"} Nov 02 17:39:37.833083 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:37.833026 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podStartSLOduration=1.8330108699999998 podStartE2EDuration="1.83301087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:37.83105727 +0000 UTC m=+596.452424818" watchObservedRunningTime="2026-11-02 17:39:37.83301087 +0000 UTC m=+596.454370645" Nov 02 17:39:38.324315 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:38.324284 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:41.564505 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:41.564475 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 17:39:41.568410 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:41.568388 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 17:39:41.580883 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:41.580860 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 17:39:41.586717 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:41.586692 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 17:39:47.324373 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:47.324326 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:47.337359 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:47.337328 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:47.866362 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:47.866329 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:39:50.423498 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.423456 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6"] Nov 02 17:39:50.429499 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.429476 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:50.447008 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.446974 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6"] Nov 02 17:39:50.453935 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.453894 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f0a3b8\"/\"kube-root-ca.crt\"" Nov 02 17:39:50.456789 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.456765 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f0a3b8\"/\"openshift-service-ca.crt\"" Nov 02 17:39:50.457018 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.457005 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f0a3b8\"/\"default-dockercfg-4zkbn\"" Nov 02 17:39:50.457071 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.457035 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f0a3b8\"/\"premium-w2-f0a3b8-kserve-self-signed-certs\"" Nov 02 17:39:50.528072 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.528040 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78574547-89da-456a-a2ca-137fba06a5b3-tls-certs\") pod \"premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:50.528300 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.528087 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-dshm\") pod \"premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:50.528300 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.528109 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkhbm\" (UniqueName: \"kubernetes.io/projected/78574547-89da-456a-a2ca-137fba06a5b3-kube-api-access-kkhbm\") pod \"premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:50.528300 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.528193 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-model-cache\") pod \"premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:50.528300 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.528249 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-tmp-dir\") pod \"premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:50.528531 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.528476 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-home\") pod \"premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:50.629352 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.629307 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-home\") pod \"premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:50.629564 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.629383 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78574547-89da-456a-a2ca-137fba06a5b3-tls-certs\") pod \"premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:50.629709 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.629685 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-dshm\") pod \"premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:50.629778 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.629726 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhbm\" (UniqueName: \"kubernetes.io/projected/78574547-89da-456a-a2ca-137fba06a5b3-kube-api-access-kkhbm\") pod \"premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:50.629778 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.629761 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-model-cache\") pod \"premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:50.629897 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.629807 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-tmp-dir\") pod \"premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:50.630485 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.630457 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-model-cache\") pod \"premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:50.630982 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.630945 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-tmp-dir\") pod \"premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:50.631160 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.630992 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-home\") pod \"premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:50.632667 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.632643 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78574547-89da-456a-a2ca-137fba06a5b3-tls-certs\") pod \"premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:50.633921 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.633899 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-dshm\") pod \"premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:50.654614 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.654582 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkhbm\" (UniqueName: \"kubernetes.io/projected/78574547-89da-456a-a2ca-137fba06a5b3-kube-api-access-kkhbm\") pod \"premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:50.743961 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:50.743878 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:39:51.046629 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:51.046596 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6"] Nov 02 17:39:51.047095 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:39:51.047055 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78574547_89da_456a_a2ca_137fba06a5b3.slice/crio-befa37bde5e78a8de8ee488a437bf555477a6e555545a282f4241235749e8dc7 WatchSource:0}: Error finding container befa37bde5e78a8de8ee488a437bf555477a6e555545a282f4241235749e8dc7: Status 404 returned error can't find the container with id befa37bde5e78a8de8ee488a437bf555477a6e555545a282f4241235749e8dc7 Nov 02 17:39:51.769229 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:39:51.769197 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:39:51.885567 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:51.885527 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" event={"ID":"78574547-89da-456a-a2ca-137fba06a5b3","Type":"ContainerStarted","Data":"e75c11d78793d51129b7f3f43de7e6bd75b2c3536ba35bebad2d80534aa3d32f"} Nov 02 17:39:51.885567 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:51.885577 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" event={"ID":"78574547-89da-456a-a2ca-137fba06a5b3","Type":"ContainerStarted","Data":"befa37bde5e78a8de8ee488a437bf555477a6e555545a282f4241235749e8dc7"} Nov 02 17:39:51.929445 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:51.929396 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podStartSLOduration=1.929383256 podStartE2EDuration="1.929383256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:51.926962999 +0000 UTC m=+610.548322777" watchObservedRunningTime="2026-11-02 17:39:51.929383256 +0000 UTC m=+610.550743031" Nov 02 17:39:52.744353 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:39:52.744314 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:40:00.744210 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:00.744169 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:40:00.757075 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:00.757052 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:40:00.960540 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:00.960505 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:40:21.770682 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:40:21.770646 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:40:25.128860 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:25.128812 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts"] Nov 02 17:40:25.134739 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:25.134718 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts" Nov 02 17:40:25.153144 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:25.153120 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts"] Nov 02 17:40:25.160948 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:25.160916 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:40:25.161055 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:25.160959 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:40:25.284620 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:25.284586 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1def7ef1-2e02-4174-8b07-e418879e3cdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf07803e-grrts\" (UID: \"1def7ef1-2e02-4174-8b07-e418879e3cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts" Nov 02 17:40:25.284816 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:25.284750 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmwqd\" (UniqueName: \"kubernetes.io/projected/1def7ef1-2e02-4174-8b07-e418879e3cdc-kube-api-access-hmwqd\") pod \"maas-api-revoke-sub-cf07803e-grrts\" (UID: \"1def7ef1-2e02-4174-8b07-e418879e3cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts" Nov 02 17:40:25.284928 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:25.284823 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1def7ef1-2e02-4174-8b07-e418879e3cdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf07803e-grrts\" (UID: \"1def7ef1-2e02-4174-8b07-e418879e3cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts" Nov 02 17:40:25.386153 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:25.386070 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1def7ef1-2e02-4174-8b07-e418879e3cdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf07803e-grrts\" (UID: \"1def7ef1-2e02-4174-8b07-e418879e3cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts" Nov 02 17:40:25.386303 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:25.386183 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmwqd\" (UniqueName: \"kubernetes.io/projected/1def7ef1-2e02-4174-8b07-e418879e3cdc-kube-api-access-hmwqd\") pod \"maas-api-revoke-sub-cf07803e-grrts\" (UID: \"1def7ef1-2e02-4174-8b07-e418879e3cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts" Nov 02 17:40:25.386303 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:25.386223 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1def7ef1-2e02-4174-8b07-e418879e3cdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf07803e-grrts\" (UID: \"1def7ef1-2e02-4174-8b07-e418879e3cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts" Nov 02 17:40:25.386858 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:25.386827 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1def7ef1-2e02-4174-8b07-e418879e3cdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf07803e-grrts\" (UID: \"1def7ef1-2e02-4174-8b07-e418879e3cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts" Nov 02 17:40:25.388476 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:25.388458 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1def7ef1-2e02-4174-8b07-e418879e3cdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf07803e-grrts\" (UID: \"1def7ef1-2e02-4174-8b07-e418879e3cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts" Nov 02 17:40:25.410026 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:25.410001 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmwqd\" (UniqueName: \"kubernetes.io/projected/1def7ef1-2e02-4174-8b07-e418879e3cdc-kube-api-access-hmwqd\") pod \"maas-api-revoke-sub-cf07803e-grrts\" (UID: \"1def7ef1-2e02-4174-8b07-e418879e3cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts" Nov 02 17:40:25.448967 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:25.448937 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts" Nov 02 17:40:25.750308 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:25.750256 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts"] Nov 02 17:40:25.751363 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:40:25.751336 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1def7ef1_2e02_4174_8b07_e418879e3cdc.slice/crio-9ea536841ffa1cc9c37401362004445d7dc27b1406c46a7047093196c787595a WatchSource:0}: Error finding container 9ea536841ffa1cc9c37401362004445d7dc27b1406c46a7047093196c787595a: Status 404 returned error can't find the container with id 9ea536841ffa1cc9c37401362004445d7dc27b1406c46a7047093196c787595a Nov 02 17:40:25.753376 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:25.753358 4860 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:40:26.113455 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:26.113417 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts" event={"ID":"1def7ef1-2e02-4174-8b07-e418879e3cdc","Type":"ContainerStarted","Data":"9ea536841ffa1cc9c37401362004445d7dc27b1406c46a7047093196c787595a"} Nov 02 17:40:37.187832 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:37.187792 4860 generic.go:356] "Generic (PLEG): container finished" podID="1def7ef1-2e02-4174-8b07-e418879e3cdc" containerID="f063160d7576a291bccc11fa9832ea699738dfabfd13cb5abe5856813fd04500" exitCode=0 Nov 02 17:40:37.188239 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:37.187889 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts" event={"ID":"1def7ef1-2e02-4174-8b07-e418879e3cdc","Type":"ContainerDied","Data":"f063160d7576a291bccc11fa9832ea699738dfabfd13cb5abe5856813fd04500"} Nov 02 17:40:38.340457 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:38.340430 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts" Nov 02 17:40:38.524721 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:38.524629 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1def7ef1-2e02-4174-8b07-e418879e3cdc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1def7ef1-2e02-4174-8b07-e418879e3cdc-maas-api-xks-ca\") pod \"1def7ef1-2e02-4174-8b07-e418879e3cdc\" (UID: \"1def7ef1-2e02-4174-8b07-e418879e3cdc\") " Nov 02 17:40:38.524934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:38.524753 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1def7ef1-2e02-4174-8b07-e418879e3cdc-kube-api-access-hmwqd\" (UniqueName: \"kubernetes.io/projected/1def7ef1-2e02-4174-8b07-e418879e3cdc-kube-api-access-hmwqd\") pod \"1def7ef1-2e02-4174-8b07-e418879e3cdc\" (UID: \"1def7ef1-2e02-4174-8b07-e418879e3cdc\") " Nov 02 17:40:38.524934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:38.524816 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1def7ef1-2e02-4174-8b07-e418879e3cdc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1def7ef1-2e02-4174-8b07-e418879e3cdc-maas-api-openshift-ca\") pod \"1def7ef1-2e02-4174-8b07-e418879e3cdc\" (UID: \"1def7ef1-2e02-4174-8b07-e418879e3cdc\") " Nov 02 17:40:38.525200 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:38.525164 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1def7ef1-2e02-4174-8b07-e418879e3cdc-maas-api-openshift-ca" pod "1def7ef1-2e02-4174-8b07-e418879e3cdc" (UID: "1def7ef1-2e02-4174-8b07-e418879e3cdc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:40:38.527012 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:38.526988 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1def7ef1-2e02-4174-8b07-e418879e3cdc-maas-api-xks-ca" pod "1def7ef1-2e02-4174-8b07-e418879e3cdc" (UID: "1def7ef1-2e02-4174-8b07-e418879e3cdc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:40:38.527183 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:38.527161 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1def7ef1-2e02-4174-8b07-e418879e3cdc-kube-api-access-hmwqd" pod "1def7ef1-2e02-4174-8b07-e418879e3cdc" (UID: "1def7ef1-2e02-4174-8b07-e418879e3cdc"). InnerVolumeSpecName "kube-api-access-hmwqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:40:38.626629 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:38.626588 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1def7ef1-2e02-4174-8b07-e418879e3cdc-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:40:38.626629 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:38.626626 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1def7ef1-2e02-4174-8b07-e418879e3cdc-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:40:38.626829 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:38.626641 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmwqd\" (UniqueName: \"kubernetes.io/projected/1def7ef1-2e02-4174-8b07-e418879e3cdc-kube-api-access-hmwqd\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:40:39.202225 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:39.202192 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts" Nov 02 17:40:39.202410 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:39.202234 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts" event={"ID":"1def7ef1-2e02-4174-8b07-e418879e3cdc","Type":"ContainerDied","Data":"9ea536841ffa1cc9c37401362004445d7dc27b1406c46a7047093196c787595a"} Nov 02 17:40:39.202410 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:39.202291 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ea536841ffa1cc9c37401362004445d7dc27b1406c46a7047093196c787595a" Nov 02 17:40:42.042263 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:42.042229 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml"] Nov 02 17:40:42.042770 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:42.042756 4860 cpu_manager.go:414] "containerMap: removing container" podUID="1def7ef1-2e02-4174-8b07-e418879e3cdc" containerName="revoke-keys" Nov 02 17:40:42.042813 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:42.042770 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1def7ef1-2e02-4174-8b07-e418879e3cdc" containerName="revoke-keys" Nov 02 17:40:42.042921 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:42.042861 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="1def7ef1-2e02-4174-8b07-e418879e3cdc" containerName="revoke-keys" Nov 02 17:40:42.046285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:42.046268 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml" Nov 02 17:40:42.058818 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:42.058786 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pwd5\" (UniqueName: \"kubernetes.io/projected/c7b0417d-52a1-4c27-941d-da74c9d5de0d-kube-api-access-5pwd5\") pod \"maas-api-revoke-sub-0bf84112-9wlml\" (UID: \"c7b0417d-52a1-4c27-941d-da74c9d5de0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml" Nov 02 17:40:42.059071 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:42.058918 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b0417d-52a1-4c27-941d-da74c9d5de0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bf84112-9wlml\" (UID: \"c7b0417d-52a1-4c27-941d-da74c9d5de0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml" Nov 02 17:40:42.059071 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:42.058985 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b0417d-52a1-4c27-941d-da74c9d5de0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bf84112-9wlml\" (UID: \"c7b0417d-52a1-4c27-941d-da74c9d5de0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml" Nov 02 17:40:42.069756 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:42.069724 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml"] Nov 02 17:40:42.076488 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:42.076456 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:40:42.076621 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:42.076516 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:40:42.160308 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:42.160266 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b0417d-52a1-4c27-941d-da74c9d5de0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bf84112-9wlml\" (UID: \"c7b0417d-52a1-4c27-941d-da74c9d5de0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml" Nov 02 17:40:42.160490 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:42.160335 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b0417d-52a1-4c27-941d-da74c9d5de0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bf84112-9wlml\" (UID: \"c7b0417d-52a1-4c27-941d-da74c9d5de0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml" Nov 02 17:40:42.160490 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:42.160444 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pwd5\" (UniqueName: \"kubernetes.io/projected/c7b0417d-52a1-4c27-941d-da74c9d5de0d-kube-api-access-5pwd5\") pod \"maas-api-revoke-sub-0bf84112-9wlml\" (UID: \"c7b0417d-52a1-4c27-941d-da74c9d5de0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml" Nov 02 17:40:42.160963 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:42.160943 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b0417d-52a1-4c27-941d-da74c9d5de0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bf84112-9wlml\" (UID: \"c7b0417d-52a1-4c27-941d-da74c9d5de0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml" Nov 02 17:40:42.162674 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:42.162655 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b0417d-52a1-4c27-941d-da74c9d5de0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bf84112-9wlml\" (UID: \"c7b0417d-52a1-4c27-941d-da74c9d5de0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml" Nov 02 17:40:42.184746 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:42.184724 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pwd5\" (UniqueName: \"kubernetes.io/projected/c7b0417d-52a1-4c27-941d-da74c9d5de0d-kube-api-access-5pwd5\") pod \"maas-api-revoke-sub-0bf84112-9wlml\" (UID: \"c7b0417d-52a1-4c27-941d-da74c9d5de0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml" Nov 02 17:40:42.362976 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:42.362938 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml" Nov 02 17:40:42.664714 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:42.664684 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml"] Nov 02 17:40:42.665642 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:40:42.665618 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b0417d_52a1_4c27_941d_da74c9d5de0d.slice/crio-4e99b280522730100012075d09315bb019d75763730cdc329091e3477b39ca72 WatchSource:0}: Error finding container 4e99b280522730100012075d09315bb019d75763730cdc329091e3477b39ca72: Status 404 returned error can't find the container with id 4e99b280522730100012075d09315bb019d75763730cdc329091e3477b39ca72 Nov 02 17:40:43.235211 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:43.235116 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml" event={"ID":"c7b0417d-52a1-4c27-941d-da74c9d5de0d","Type":"ContainerStarted","Data":"37dac0de009ad39b8c96b873da9167a5a442b3e91f34f6bbda82d046722ff57a"} Nov 02 17:40:43.235211 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:43.235179 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml" event={"ID":"c7b0417d-52a1-4c27-941d-da74c9d5de0d","Type":"ContainerStarted","Data":"4e99b280522730100012075d09315bb019d75763730cdc329091e3477b39ca72"} Nov 02 17:40:43.279496 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:43.279439 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml" podStartSLOduration=1.279422364 podStartE2EDuration="1.279422364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:43.276110756 +0000 UTC m=+661.897470533" watchObservedRunningTime="2026-11-02 17:40:43.279422364 +0000 UTC m=+661.900782141" Nov 02 17:40:44.243297 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:44.243264 4860 generic.go:356] "Generic (PLEG): container finished" podID="c7b0417d-52a1-4c27-941d-da74c9d5de0d" containerID="37dac0de009ad39b8c96b873da9167a5a442b3e91f34f6bbda82d046722ff57a" exitCode=0 Nov 02 17:40:44.243667 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:44.243350 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml" event={"ID":"c7b0417d-52a1-4c27-941d-da74c9d5de0d","Type":"ContainerDied","Data":"37dac0de009ad39b8c96b873da9167a5a442b3e91f34f6bbda82d046722ff57a"} Nov 02 17:40:45.410988 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:45.410963 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml" Nov 02 17:40:45.492237 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:45.492205 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7b0417d-52a1-4c27-941d-da74c9d5de0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b0417d-52a1-4c27-941d-da74c9d5de0d-maas-api-xks-ca\") pod \"c7b0417d-52a1-4c27-941d-da74c9d5de0d\" (UID: \"c7b0417d-52a1-4c27-941d-da74c9d5de0d\") " Nov 02 17:40:45.492535 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:45.492289 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b0417d-52a1-4c27-941d-da74c9d5de0d-kube-api-access-5pwd5\" (UniqueName: \"kubernetes.io/projected/c7b0417d-52a1-4c27-941d-da74c9d5de0d-kube-api-access-5pwd5\") pod \"c7b0417d-52a1-4c27-941d-da74c9d5de0d\" (UID: \"c7b0417d-52a1-4c27-941d-da74c9d5de0d\") " Nov 02 17:40:45.492535 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:45.492343 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b0417d-52a1-4c27-941d-da74c9d5de0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b0417d-52a1-4c27-941d-da74c9d5de0d-maas-api-openshift-ca\") pod \"c7b0417d-52a1-4c27-941d-da74c9d5de0d\" (UID: \"c7b0417d-52a1-4c27-941d-da74c9d5de0d\") " Nov 02 17:40:45.492753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:45.492723 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b0417d-52a1-4c27-941d-da74c9d5de0d-maas-api-openshift-ca" pod "c7b0417d-52a1-4c27-941d-da74c9d5de0d" (UID: "c7b0417d-52a1-4c27-941d-da74c9d5de0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:40:45.494365 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:45.494338 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7b0417d-52a1-4c27-941d-da74c9d5de0d-maas-api-xks-ca" pod "c7b0417d-52a1-4c27-941d-da74c9d5de0d" (UID: "c7b0417d-52a1-4c27-941d-da74c9d5de0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:40:45.494487 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:45.494412 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b0417d-52a1-4c27-941d-da74c9d5de0d-kube-api-access-5pwd5" pod "c7b0417d-52a1-4c27-941d-da74c9d5de0d" (UID: "c7b0417d-52a1-4c27-941d-da74c9d5de0d"). InnerVolumeSpecName "kube-api-access-5pwd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:40:45.593771 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:45.593739 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b0417d-52a1-4c27-941d-da74c9d5de0d-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:40:45.593771 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:45.593765 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b0417d-52a1-4c27-941d-da74c9d5de0d-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:40:45.593771 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:45.593776 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pwd5\" (UniqueName: \"kubernetes.io/projected/c7b0417d-52a1-4c27-941d-da74c9d5de0d-kube-api-access-5pwd5\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:40:46.257190 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:46.257159 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml" Nov 02 17:40:46.257363 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:46.257198 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml" event={"ID":"c7b0417d-52a1-4c27-941d-da74c9d5de0d","Type":"ContainerDied","Data":"4e99b280522730100012075d09315bb019d75763730cdc329091e3477b39ca72"} Nov 02 17:40:46.257363 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:46.257244 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e99b280522730100012075d09315bb019d75763730cdc329091e3477b39ca72" Nov 02 17:40:51.770217 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:40:51.770180 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:40:57.011358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:57.011280 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l"] Nov 02 17:40:57.011793 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:57.011780 4860 cpu_manager.go:414] "containerMap: removing container" podUID="c7b0417d-52a1-4c27-941d-da74c9d5de0d" containerName="revoke-keys" Nov 02 17:40:57.011833 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:57.011794 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b0417d-52a1-4c27-941d-da74c9d5de0d" containerName="revoke-keys" Nov 02 17:40:57.011936 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:57.011906 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b0417d-52a1-4c27-941d-da74c9d5de0d" containerName="revoke-keys" Nov 02 17:40:57.015455 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:57.015433 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l" Nov 02 17:40:57.035563 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:57.035536 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l"] Nov 02 17:40:57.044210 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:57.044185 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:40:57.044210 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:57.044191 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:40:57.115167 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:57.115130 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgvmn\" (UniqueName: \"kubernetes.io/projected/7b0c5a95-b8a3-44f2-a11f-11978c76a712-kube-api-access-jgvmn\") pod \"maas-api-revoke-sub-d60a57f8-5gg4l\" (UID: \"7b0c5a95-b8a3-44f2-a11f-11978c76a712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l" Nov 02 17:40:57.115340 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:57.115200 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0c5a95-b8a3-44f2-a11f-11978c76a712-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d60a57f8-5gg4l\" (UID: \"7b0c5a95-b8a3-44f2-a11f-11978c76a712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l" Nov 02 17:40:57.115340 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:57.115321 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0c5a95-b8a3-44f2-a11f-11978c76a712-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d60a57f8-5gg4l\" (UID: \"7b0c5a95-b8a3-44f2-a11f-11978c76a712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l" Nov 02 17:40:57.216578 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:57.216527 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0c5a95-b8a3-44f2-a11f-11978c76a712-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d60a57f8-5gg4l\" (UID: \"7b0c5a95-b8a3-44f2-a11f-11978c76a712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l" Nov 02 17:40:57.216743 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:57.216625 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0c5a95-b8a3-44f2-a11f-11978c76a712-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d60a57f8-5gg4l\" (UID: \"7b0c5a95-b8a3-44f2-a11f-11978c76a712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l" Nov 02 17:40:57.216743 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:57.216684 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgvmn\" (UniqueName: \"kubernetes.io/projected/7b0c5a95-b8a3-44f2-a11f-11978c76a712-kube-api-access-jgvmn\") pod \"maas-api-revoke-sub-d60a57f8-5gg4l\" (UID: \"7b0c5a95-b8a3-44f2-a11f-11978c76a712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l" Nov 02 17:40:57.217206 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:57.217174 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0c5a95-b8a3-44f2-a11f-11978c76a712-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d60a57f8-5gg4l\" (UID: \"7b0c5a95-b8a3-44f2-a11f-11978c76a712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l" Nov 02 17:40:57.219202 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:57.219172 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0c5a95-b8a3-44f2-a11f-11978c76a712-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d60a57f8-5gg4l\" (UID: \"7b0c5a95-b8a3-44f2-a11f-11978c76a712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l" Nov 02 17:40:57.240326 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:57.240296 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgvmn\" (UniqueName: \"kubernetes.io/projected/7b0c5a95-b8a3-44f2-a11f-11978c76a712-kube-api-access-jgvmn\") pod \"maas-api-revoke-sub-d60a57f8-5gg4l\" (UID: \"7b0c5a95-b8a3-44f2-a11f-11978c76a712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l" Nov 02 17:40:57.329111 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:57.329073 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l" Nov 02 17:40:57.637575 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:57.637541 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l"] Nov 02 17:40:57.637731 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:40:57.637580 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0c5a95_b8a3_44f2_a11f_11978c76a712.slice/crio-b92ca5eb885e183512d25a3c18f5c2af2d8786b6ce55f2302a01871eff758cf3 WatchSource:0}: Error finding container b92ca5eb885e183512d25a3c18f5c2af2d8786b6ce55f2302a01871eff758cf3: Status 404 returned error can't find the container with id b92ca5eb885e183512d25a3c18f5c2af2d8786b6ce55f2302a01871eff758cf3 Nov 02 17:40:58.347276 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:58.347241 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l" event={"ID":"7b0c5a95-b8a3-44f2-a11f-11978c76a712","Type":"ContainerStarted","Data":"a6d2d9a2613c04904534050d0257ccf348d66ef88ab61aa852183ead28dfd968"} Nov 02 17:40:58.347643 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:58.347293 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l" event={"ID":"7b0c5a95-b8a3-44f2-a11f-11978c76a712","Type":"ContainerStarted","Data":"b92ca5eb885e183512d25a3c18f5c2af2d8786b6ce55f2302a01871eff758cf3"} Nov 02 17:40:58.388217 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:58.388164 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l" podStartSLOduration=2.388150713 podStartE2EDuration="2.388150713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:58.38796383 +0000 UTC m=+677.009323606" watchObservedRunningTime="2026-11-02 17:40:58.388150713 +0000 UTC m=+677.009510488" Nov 02 17:40:59.355159 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:59.355117 4860 generic.go:356] "Generic (PLEG): container finished" podID="7b0c5a95-b8a3-44f2-a11f-11978c76a712" containerID="a6d2d9a2613c04904534050d0257ccf348d66ef88ab61aa852183ead28dfd968" exitCode=0 Nov 02 17:40:59.355557 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:59.355200 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l" event={"ID":"7b0c5a95-b8a3-44f2-a11f-11978c76a712","Type":"ContainerDied","Data":"a6d2d9a2613c04904534050d0257ccf348d66ef88ab61aa852183ead28dfd968"} Nov 02 17:40:59.675123 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:59.675044 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn"] Nov 02 17:40:59.679030 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:59.679012 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn" Nov 02 17:40:59.699365 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:59.699340 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn"] Nov 02 17:40:59.745857 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:59.745806 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da516c96-b934-48dd-9806-24795e13e854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69152f6e-l28nn\" (UID: \"da516c96-b934-48dd-9806-24795e13e854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn" Nov 02 17:40:59.746019 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:59.745977 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da516c96-b934-48dd-9806-24795e13e854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69152f6e-l28nn\" (UID: \"da516c96-b934-48dd-9806-24795e13e854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn" Nov 02 17:40:59.746097 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:59.746080 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l74gx\" (UniqueName: \"kubernetes.io/projected/da516c96-b934-48dd-9806-24795e13e854-kube-api-access-l74gx\") pod \"maas-api-revoke-sub-69152f6e-l28nn\" (UID: \"da516c96-b934-48dd-9806-24795e13e854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn" Nov 02 17:40:59.846782 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:59.846743 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da516c96-b934-48dd-9806-24795e13e854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69152f6e-l28nn\" (UID: \"da516c96-b934-48dd-9806-24795e13e854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn" Nov 02 17:40:59.847003 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:59.846812 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da516c96-b934-48dd-9806-24795e13e854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69152f6e-l28nn\" (UID: \"da516c96-b934-48dd-9806-24795e13e854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn" Nov 02 17:40:59.847003 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:59.846880 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l74gx\" (UniqueName: \"kubernetes.io/projected/da516c96-b934-48dd-9806-24795e13e854-kube-api-access-l74gx\") pod \"maas-api-revoke-sub-69152f6e-l28nn\" (UID: \"da516c96-b934-48dd-9806-24795e13e854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn" Nov 02 17:40:59.847537 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:59.847511 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da516c96-b934-48dd-9806-24795e13e854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69152f6e-l28nn\" (UID: \"da516c96-b934-48dd-9806-24795e13e854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn" Nov 02 17:40:59.849348 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:59.849322 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da516c96-b934-48dd-9806-24795e13e854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69152f6e-l28nn\" (UID: \"da516c96-b934-48dd-9806-24795e13e854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn" Nov 02 17:40:59.870705 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:59.870677 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l74gx\" (UniqueName: \"kubernetes.io/projected/da516c96-b934-48dd-9806-24795e13e854-kube-api-access-l74gx\") pod \"maas-api-revoke-sub-69152f6e-l28nn\" (UID: \"da516c96-b934-48dd-9806-24795e13e854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn" Nov 02 17:40:59.993222 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:40:59.993121 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn" Nov 02 17:41:00.305133 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.305101 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn"] Nov 02 17:41:00.305919 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:41:00.305839 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda516c96_b934_48dd_9806_24795e13e854.slice/crio-919d7947537fccfa9f671b404077bb01265e7cec80dd8e279e690f1ffcf20f79 WatchSource:0}: Error finding container 919d7947537fccfa9f671b404077bb01265e7cec80dd8e279e690f1ffcf20f79: Status 404 returned error can't find the container with id 919d7947537fccfa9f671b404077bb01265e7cec80dd8e279e690f1ffcf20f79 Nov 02 17:41:00.363255 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.363219 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn" event={"ID":"da516c96-b934-48dd-9806-24795e13e854","Type":"ContainerStarted","Data":"919d7947537fccfa9f671b404077bb01265e7cec80dd8e279e690f1ffcf20f79"} Nov 02 17:41:00.495027 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.494999 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l" Nov 02 17:41:00.554061 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.553985 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b0c5a95-b8a3-44f2-a11f-11978c76a712-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0c5a95-b8a3-44f2-a11f-11978c76a712-maas-api-xks-ca\") pod \"7b0c5a95-b8a3-44f2-a11f-11978c76a712\" (UID: \"7b0c5a95-b8a3-44f2-a11f-11978c76a712\") " Nov 02 17:41:00.554215 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.554057 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0c5a95-b8a3-44f2-a11f-11978c76a712-kube-api-access-jgvmn\" (UniqueName: \"kubernetes.io/projected/7b0c5a95-b8a3-44f2-a11f-11978c76a712-kube-api-access-jgvmn\") pod \"7b0c5a95-b8a3-44f2-a11f-11978c76a712\" (UID: \"7b0c5a95-b8a3-44f2-a11f-11978c76a712\") " Nov 02 17:41:00.554215 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.554195 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b0c5a95-b8a3-44f2-a11f-11978c76a712-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0c5a95-b8a3-44f2-a11f-11978c76a712-maas-api-openshift-ca\") pod \"7b0c5a95-b8a3-44f2-a11f-11978c76a712\" (UID: \"7b0c5a95-b8a3-44f2-a11f-11978c76a712\") " Nov 02 17:41:00.554636 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.554595 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0c5a95-b8a3-44f2-a11f-11978c76a712-maas-api-openshift-ca" pod "7b0c5a95-b8a3-44f2-a11f-11978c76a712" (UID: "7b0c5a95-b8a3-44f2-a11f-11978c76a712"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:00.556486 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.556456 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0c5a95-b8a3-44f2-a11f-11978c76a712-maas-api-xks-ca" pod "7b0c5a95-b8a3-44f2-a11f-11978c76a712" (UID: "7b0c5a95-b8a3-44f2-a11f-11978c76a712"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:00.556591 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.556509 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0c5a95-b8a3-44f2-a11f-11978c76a712-kube-api-access-jgvmn" pod "7b0c5a95-b8a3-44f2-a11f-11978c76a712" (UID: "7b0c5a95-b8a3-44f2-a11f-11978c76a712"). InnerVolumeSpecName "kube-api-access-jgvmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:00.599842 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.599809 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf"] Nov 02 17:41:00.600359 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.600344 4860 cpu_manager.go:414] "containerMap: removing container" podUID="7b0c5a95-b8a3-44f2-a11f-11978c76a712" containerName="revoke-keys" Nov 02 17:41:00.600411 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.600361 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0c5a95-b8a3-44f2-a11f-11978c76a712" containerName="revoke-keys" Nov 02 17:41:00.600464 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.600455 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0c5a95-b8a3-44f2-a11f-11978c76a712" containerName="revoke-keys" Nov 02 17:41:00.603690 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.603670 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.618783 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.618759 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-7df526-openshift-default-dockercfg-cjrwk\"" Nov 02 17:41:00.628514 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.628478 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf"] Nov 02 17:41:00.655704 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.655673 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgvmn\" (UniqueName: \"kubernetes.io/projected/7b0c5a95-b8a3-44f2-a11f-11978c76a712-kube-api-access-jgvmn\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:00.655704 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.655704 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0c5a95-b8a3-44f2-a11f-11978c76a712-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:00.655949 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.655718 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0c5a95-b8a3-44f2-a11f-11978c76a712-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:00.753173 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.753142 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz"] Nov 02 17:41:00.756413 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.756378 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.756543 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.756445 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.756543 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.756510 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.756543 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.756537 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc2pw\" (UniqueName: \"kubernetes.io/projected/b53fb7b2-d91c-409b-bf83-e24b95a7608f-kube-api-access-mc2pw\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.756662 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.756577 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.756662 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.756609 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.756726 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.756698 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.756726 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.756718 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.756790 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.756737 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.757304 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.757284 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.781033 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.781006 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-120394-openshift-default-dockercfg-qq496\"" Nov 02 17:41:00.781186 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.781075 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz"] Nov 02 17:41:00.858406 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.858371 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpx6v\" (UniqueName: \"kubernetes.io/projected/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-kube-api-access-hpx6v\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.858571 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.858487 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.858571 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.858519 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.858571 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.858566 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.858770 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.858602 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.858770 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.858730 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.858906 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.858768 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.858906 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.858807 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.858906 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.858834 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.858906 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.858889 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc2pw\" (UniqueName: \"kubernetes.io/projected/b53fb7b2-d91c-409b-bf83-e24b95a7608f-kube-api-access-mc2pw\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.859129 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.858904 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.859129 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.858919 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.859250 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.859195 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.859480 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.859458 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.859636 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.859550 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.859951 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.859915 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.860041 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.859967 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.860041 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.860023 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.860164 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.860040 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.860164 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.860069 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.860164 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.860102 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.860164 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.860133 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.860740 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.860715 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.862445 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.862415 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.862770 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.862746 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.882562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.882541 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.882908 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.882888 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc2pw\" (UniqueName: \"kubernetes.io/projected/b53fb7b2-d91c-409b-bf83-e24b95a7608f-kube-api-access-mc2pw\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.919053 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.919027 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:00.961400 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.961367 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.961582 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.961436 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpx6v\" (UniqueName: \"kubernetes.io/projected/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-kube-api-access-hpx6v\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.961582 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.961475 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.961582 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.961502 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.962047 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.961607 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.962047 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.961653 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.962047 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.961692 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.962047 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.961723 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.962047 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.961777 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.962047 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.961894 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.962425 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.962082 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.962478 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.962420 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.962478 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.962455 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.962737 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.962714 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.963782 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.963755 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.964056 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.964037 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.986743 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.986710 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpx6v\" (UniqueName: \"kubernetes.io/projected/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-kube-api-access-hpx6v\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:00.986974 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:00.986950 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:01.071080 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.071039 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:01.266268 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.266235 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l"] Nov 02 17:41:01.270519 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.270491 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l" Nov 02 17:41:01.278403 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.278378 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf"] Nov 02 17:41:01.280037 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.279995 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:41:01.280136 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.280075 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:41:01.280136 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.280104 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:41:01.293294 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.293259 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l"] Nov 02 17:41:01.367154 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.367122 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547cae52-11eb-4166-b466-6cf0528e2ce2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d38be61d-7rb5l\" (UID: \"547cae52-11eb-4166-b466-6cf0528e2ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l" Nov 02 17:41:01.367492 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.367323 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547cae52-11eb-4166-b466-6cf0528e2ce2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d38be61d-7rb5l\" (UID: \"547cae52-11eb-4166-b466-6cf0528e2ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l" Nov 02 17:41:01.367492 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.367369 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85jsb\" (UniqueName: \"kubernetes.io/projected/547cae52-11eb-4166-b466-6cf0528e2ce2-kube-api-access-85jsb\") pod \"maas-api-revoke-sub-d38be61d-7rb5l\" (UID: \"547cae52-11eb-4166-b466-6cf0528e2ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l" Nov 02 17:41:01.371258 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.371216 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l" event={"ID":"7b0c5a95-b8a3-44f2-a11f-11978c76a712","Type":"ContainerDied","Data":"b92ca5eb885e183512d25a3c18f5c2af2d8786b6ce55f2302a01871eff758cf3"} Nov 02 17:41:01.371414 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.371324 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b92ca5eb885e183512d25a3c18f5c2af2d8786b6ce55f2302a01871eff758cf3" Nov 02 17:41:01.371414 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.371233 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l" Nov 02 17:41:01.376682 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.376610 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" event={"ID":"b53fb7b2-d91c-409b-bf83-e24b95a7608f","Type":"ContainerStarted","Data":"30faeedd7646bee6bf8c2c08acf678ef95aee941ff9af52570e2eb64031430ad"} Nov 02 17:41:01.376682 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.376660 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" event={"ID":"b53fb7b2-d91c-409b-bf83-e24b95a7608f","Type":"ContainerStarted","Data":"b4e40b546ae45993bb243ebce589b80d14a8208e0bb0ea298c1e0d01f9fc0ae4"} Nov 02 17:41:01.378756 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.378729 4860 generic.go:356] "Generic (PLEG): container finished" podID="da516c96-b934-48dd-9806-24795e13e854" containerID="1baf86abe78610217575dd1dcb525ff5e948e9acc2abfa37af064de7c42a85a0" exitCode=0 Nov 02 17:41:01.378902 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.378823 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn" event={"ID":"da516c96-b934-48dd-9806-24795e13e854","Type":"ContainerDied","Data":"1baf86abe78610217575dd1dcb525ff5e948e9acc2abfa37af064de7c42a85a0"} Nov 02 17:41:01.389117 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:41:01.389085 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2cf636_17c0_4940_84d5_4e7f9faa7dc9.slice/crio-9577d38f372cc2904e2adb171b9ce9cb5fb90bf66d632c03e6eecf4c127aa254 WatchSource:0}: Error finding container 9577d38f372cc2904e2adb171b9ce9cb5fb90bf66d632c03e6eecf4c127aa254: Status 404 returned error can't find the container with id 9577d38f372cc2904e2adb171b9ce9cb5fb90bf66d632c03e6eecf4c127aa254 Nov 02 17:41:01.390751 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.390722 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz"] Nov 02 17:41:01.391899 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.391867 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:41:01.392033 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.391930 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:41:01.392033 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.391963 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:41:01.432636 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.432587 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" podStartSLOduration=1.432573142 podStartE2EDuration="1.432573142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:01.431442956 +0000 UTC m=+680.052802733" watchObservedRunningTime="2026-11-02 17:41:01.432573142 +0000 UTC m=+680.053932919" Nov 02 17:41:01.468906 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.468872 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547cae52-11eb-4166-b466-6cf0528e2ce2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d38be61d-7rb5l\" (UID: \"547cae52-11eb-4166-b466-6cf0528e2ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l" Nov 02 17:41:01.469107 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.468954 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85jsb\" (UniqueName: \"kubernetes.io/projected/547cae52-11eb-4166-b466-6cf0528e2ce2-kube-api-access-85jsb\") pod \"maas-api-revoke-sub-d38be61d-7rb5l\" (UID: \"547cae52-11eb-4166-b466-6cf0528e2ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l" Nov 02 17:41:01.469715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.469615 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547cae52-11eb-4166-b466-6cf0528e2ce2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d38be61d-7rb5l\" (UID: \"547cae52-11eb-4166-b466-6cf0528e2ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l" Nov 02 17:41:01.469869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.469620 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547cae52-11eb-4166-b466-6cf0528e2ce2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d38be61d-7rb5l\" (UID: \"547cae52-11eb-4166-b466-6cf0528e2ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l" Nov 02 17:41:01.472254 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.472230 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547cae52-11eb-4166-b466-6cf0528e2ce2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d38be61d-7rb5l\" (UID: \"547cae52-11eb-4166-b466-6cf0528e2ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l" Nov 02 17:41:01.497506 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.497476 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85jsb\" (UniqueName: \"kubernetes.io/projected/547cae52-11eb-4166-b466-6cf0528e2ce2-kube-api-access-85jsb\") pod \"maas-api-revoke-sub-d38be61d-7rb5l\" (UID: \"547cae52-11eb-4166-b466-6cf0528e2ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l" Nov 02 17:41:01.589762 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.589729 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l" Nov 02 17:41:01.905439 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:41:01.905382 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547cae52_11eb_4166_b466_6cf0528e2ce2.slice/crio-e83cf1854118f44ad1c7a64ead1b94d1bb2e6976b19e98a80be03f2fd68ef7e1 WatchSource:0}: Error finding container e83cf1854118f44ad1c7a64ead1b94d1bb2e6976b19e98a80be03f2fd68ef7e1: Status 404 returned error can't find the container with id e83cf1854118f44ad1c7a64ead1b94d1bb2e6976b19e98a80be03f2fd68ef7e1 Nov 02 17:41:01.905536 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.905475 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l"] Nov 02 17:41:01.919914 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:01.919891 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:02.411320 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:02.411216 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" event={"ID":"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9","Type":"ContainerStarted","Data":"9c789f0f324a1d3863d433a4bd09ad474e0df306700e59720602436ef13fc6ff"} Nov 02 17:41:02.411320 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:02.411283 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" event={"ID":"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9","Type":"ContainerStarted","Data":"9577d38f372cc2904e2adb171b9ce9cb5fb90bf66d632c03e6eecf4c127aa254"} Nov 02 17:41:02.413420 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:02.413390 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l" event={"ID":"547cae52-11eb-4166-b466-6cf0528e2ce2","Type":"ContainerStarted","Data":"e83cf1854118f44ad1c7a64ead1b94d1bb2e6976b19e98a80be03f2fd68ef7e1"} Nov 02 17:41:02.485937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:02.478181 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" podStartSLOduration=2.478163596 podStartE2EDuration="2.478163596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:02.477346664 +0000 UTC m=+681.098706465" watchObservedRunningTime="2026-11-02 17:41:02.478163596 +0000 UTC m=+681.099523375" Nov 02 17:41:02.713598 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:02.713562 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn" Nov 02 17:41:02.806927 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:02.800251 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da516c96-b934-48dd-9806-24795e13e854-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da516c96-b934-48dd-9806-24795e13e854-maas-api-xks-ca\") pod \"da516c96-b934-48dd-9806-24795e13e854\" (UID: \"da516c96-b934-48dd-9806-24795e13e854\") " Nov 02 17:41:02.806927 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:02.800368 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da516c96-b934-48dd-9806-24795e13e854-kube-api-access-l74gx\" (UniqueName: \"kubernetes.io/projected/da516c96-b934-48dd-9806-24795e13e854-kube-api-access-l74gx\") pod \"da516c96-b934-48dd-9806-24795e13e854\" (UID: \"da516c96-b934-48dd-9806-24795e13e854\") " Nov 02 17:41:02.806927 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:02.800408 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da516c96-b934-48dd-9806-24795e13e854-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da516c96-b934-48dd-9806-24795e13e854-maas-api-openshift-ca\") pod \"da516c96-b934-48dd-9806-24795e13e854\" (UID: \"da516c96-b934-48dd-9806-24795e13e854\") " Nov 02 17:41:02.806927 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:02.805370 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da516c96-b934-48dd-9806-24795e13e854-maas-api-openshift-ca" pod "da516c96-b934-48dd-9806-24795e13e854" (UID: "da516c96-b934-48dd-9806-24795e13e854"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:02.819659 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:02.819599 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da516c96-b934-48dd-9806-24795e13e854-maas-api-xks-ca" pod "da516c96-b934-48dd-9806-24795e13e854" (UID: "da516c96-b934-48dd-9806-24795e13e854"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:02.820676 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:02.820597 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da516c96-b934-48dd-9806-24795e13e854-kube-api-access-l74gx" pod "da516c96-b934-48dd-9806-24795e13e854" (UID: "da516c96-b934-48dd-9806-24795e13e854"). InnerVolumeSpecName "kube-api-access-l74gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:02.909449 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:02.909414 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da516c96-b934-48dd-9806-24795e13e854-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:02.909791 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:02.909771 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l74gx\" (UniqueName: \"kubernetes.io/projected/da516c96-b934-48dd-9806-24795e13e854-kube-api-access-l74gx\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:02.909967 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:02.909953 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da516c96-b934-48dd-9806-24795e13e854-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:02.921544 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:02.921516 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 17:41:02.921776 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:02.921563 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" podUID="b53fb7b2-d91c-409b-bf83-e24b95a7608f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 17:41:03.079935 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:03.077914 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:03.099874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:03.099731 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 17:41:03.099874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:03.099812 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" podUID="9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 17:41:03.421299 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:03.421217 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn" Nov 02 17:41:03.421662 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:03.421211 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn" event={"ID":"da516c96-b934-48dd-9806-24795e13e854","Type":"ContainerDied","Data":"919d7947537fccfa9f671b404077bb01265e7cec80dd8e279e690f1ffcf20f79"} Nov 02 17:41:03.421662 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:03.421364 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="919d7947537fccfa9f671b404077bb01265e7cec80dd8e279e690f1ffcf20f79" Nov 02 17:41:03.422895 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:03.422868 4860 generic.go:356] "Generic (PLEG): container finished" podID="547cae52-11eb-4166-b466-6cf0528e2ce2" containerID="9fb1c4c845ee9f35a164cd0957604eb057b15fabe327ee45e9398829c3122749" exitCode=0 Nov 02 17:41:03.423073 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:03.422934 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l" event={"ID":"547cae52-11eb-4166-b466-6cf0528e2ce2","Type":"ContainerDied","Data":"9fb1c4c845ee9f35a164cd0957604eb057b15fabe327ee45e9398829c3122749"} Nov 02 17:41:03.919864 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:03.919823 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 17:41:03.920042 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:03.919889 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" podUID="b53fb7b2-d91c-409b-bf83-e24b95a7608f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 17:41:04.075874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:04.075360 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 17:41:04.075874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:04.075428 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" podUID="9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 17:41:04.715053 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:04.714968 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l" Nov 02 17:41:04.833743 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:04.833703 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/547cae52-11eb-4166-b466-6cf0528e2ce2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547cae52-11eb-4166-b466-6cf0528e2ce2-maas-api-xks-ca\") pod \"547cae52-11eb-4166-b466-6cf0528e2ce2\" (UID: \"547cae52-11eb-4166-b466-6cf0528e2ce2\") " Nov 02 17:41:04.833936 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:04.833837 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/547cae52-11eb-4166-b466-6cf0528e2ce2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547cae52-11eb-4166-b466-6cf0528e2ce2-maas-api-openshift-ca\") pod \"547cae52-11eb-4166-b466-6cf0528e2ce2\" (UID: \"547cae52-11eb-4166-b466-6cf0528e2ce2\") " Nov 02 17:41:04.834028 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:04.834011 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547cae52-11eb-4166-b466-6cf0528e2ce2-kube-api-access-85jsb\" (UniqueName: \"kubernetes.io/projected/547cae52-11eb-4166-b466-6cf0528e2ce2-kube-api-access-85jsb\") pod \"547cae52-11eb-4166-b466-6cf0528e2ce2\" (UID: \"547cae52-11eb-4166-b466-6cf0528e2ce2\") " Nov 02 17:41:04.834462 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:04.834431 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547cae52-11eb-4166-b466-6cf0528e2ce2-maas-api-openshift-ca" pod "547cae52-11eb-4166-b466-6cf0528e2ce2" (UID: "547cae52-11eb-4166-b466-6cf0528e2ce2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:04.835984 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:04.835963 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547cae52-11eb-4166-b466-6cf0528e2ce2-kube-api-access-85jsb" pod "547cae52-11eb-4166-b466-6cf0528e2ce2" (UID: "547cae52-11eb-4166-b466-6cf0528e2ce2"). InnerVolumeSpecName "kube-api-access-85jsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:04.836946 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:04.836927 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547cae52-11eb-4166-b466-6cf0528e2ce2-maas-api-xks-ca" pod "547cae52-11eb-4166-b466-6cf0528e2ce2" (UID: "547cae52-11eb-4166-b466-6cf0528e2ce2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:04.920257 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:04.920218 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 17:41:04.920431 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:04.920279 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" podUID="b53fb7b2-d91c-409b-bf83-e24b95a7608f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 17:41:04.934961 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:04.934925 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547cae52-11eb-4166-b466-6cf0528e2ce2-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:04.934961 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:04.934955 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85jsb\" (UniqueName: \"kubernetes.io/projected/547cae52-11eb-4166-b466-6cf0528e2ce2-kube-api-access-85jsb\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:04.935134 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:04.934970 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547cae52-11eb-4166-b466-6cf0528e2ce2-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:05.078874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:05.071556 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 17:41:05.078874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:05.071618 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" podUID="9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 17:41:05.439438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:05.439361 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l" Nov 02 17:41:05.439438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:05.439385 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l" event={"ID":"547cae52-11eb-4166-b466-6cf0528e2ce2","Type":"ContainerDied","Data":"e83cf1854118f44ad1c7a64ead1b94d1bb2e6976b19e98a80be03f2fd68ef7e1"} Nov 02 17:41:05.439652 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:05.439445 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e83cf1854118f44ad1c7a64ead1b94d1bb2e6976b19e98a80be03f2fd68ef7e1" Nov 02 17:41:05.919935 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:05.919901 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 17:41:05.920344 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:05.919952 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" podUID="b53fb7b2-d91c-409b-bf83-e24b95a7608f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 17:41:06.097875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.088000 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 17:41:06.097875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.088067 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" podUID="9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 17:41:06.243175 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.243094 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd"] Nov 02 17:41:06.243615 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.243598 4860 cpu_manager.go:414] "containerMap: removing container" podUID="547cae52-11eb-4166-b466-6cf0528e2ce2" containerName="revoke-keys" Nov 02 17:41:06.243714 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.243619 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="547cae52-11eb-4166-b466-6cf0528e2ce2" containerName="revoke-keys" Nov 02 17:41:06.243714 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.243671 4860 cpu_manager.go:414] "containerMap: removing container" podUID="da516c96-b934-48dd-9806-24795e13e854" containerName="revoke-keys" Nov 02 17:41:06.243714 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.243680 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da516c96-b934-48dd-9806-24795e13e854" containerName="revoke-keys" Nov 02 17:41:06.243808 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.243789 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="547cae52-11eb-4166-b466-6cf0528e2ce2" containerName="revoke-keys" Nov 02 17:41:06.243808 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.243801 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="da516c96-b934-48dd-9806-24795e13e854" containerName="revoke-keys" Nov 02 17:41:06.247319 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.247300 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd" Nov 02 17:41:06.279571 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.277967 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:41:06.279571 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.278578 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:41:06.289427 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.289344 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd"] Nov 02 17:41:06.350294 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.350253 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb74ceb-3aea-4cb9-b668-9d92404de76c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50a8b3e8-h2scd\" (UID: \"dfb74ceb-3aea-4cb9-b668-9d92404de76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd" Nov 02 17:41:06.350497 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.350337 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j5rr\" (UniqueName: \"kubernetes.io/projected/dfb74ceb-3aea-4cb9-b668-9d92404de76c-kube-api-access-7j5rr\") pod \"maas-api-revoke-sub-50a8b3e8-h2scd\" (UID: \"dfb74ceb-3aea-4cb9-b668-9d92404de76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd" Nov 02 17:41:06.350497 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.350394 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb74ceb-3aea-4cb9-b668-9d92404de76c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50a8b3e8-h2scd\" (UID: \"dfb74ceb-3aea-4cb9-b668-9d92404de76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd" Nov 02 17:41:06.451814 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.451784 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb74ceb-3aea-4cb9-b668-9d92404de76c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50a8b3e8-h2scd\" (UID: \"dfb74ceb-3aea-4cb9-b668-9d92404de76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd" Nov 02 17:41:06.452035 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.451836 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j5rr\" (UniqueName: \"kubernetes.io/projected/dfb74ceb-3aea-4cb9-b668-9d92404de76c-kube-api-access-7j5rr\") pod \"maas-api-revoke-sub-50a8b3e8-h2scd\" (UID: \"dfb74ceb-3aea-4cb9-b668-9d92404de76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd" Nov 02 17:41:06.452035 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.451893 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb74ceb-3aea-4cb9-b668-9d92404de76c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50a8b3e8-h2scd\" (UID: \"dfb74ceb-3aea-4cb9-b668-9d92404de76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd" Nov 02 17:41:06.452529 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.452506 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb74ceb-3aea-4cb9-b668-9d92404de76c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50a8b3e8-h2scd\" (UID: \"dfb74ceb-3aea-4cb9-b668-9d92404de76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd" Nov 02 17:41:06.499973 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.499899 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j5rr\" (UniqueName: \"kubernetes.io/projected/dfb74ceb-3aea-4cb9-b668-9d92404de76c-kube-api-access-7j5rr\") pod \"maas-api-revoke-sub-50a8b3e8-h2scd\" (UID: \"dfb74ceb-3aea-4cb9-b668-9d92404de76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd" Nov 02 17:41:06.508339 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.508279 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb74ceb-3aea-4cb9-b668-9d92404de76c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50a8b3e8-h2scd\" (UID: \"dfb74ceb-3aea-4cb9-b668-9d92404de76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd" Nov 02 17:41:06.602322 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.602282 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd" Nov 02 17:41:06.919877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.919829 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 17:41:06.920065 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:06.919893 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" podUID="b53fb7b2-d91c-409b-bf83-e24b95a7608f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 17:41:07.008938 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:07.008465 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd"] Nov 02 17:41:07.010955 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:41:07.010916 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb74ceb_3aea_4cb9_b668_9d92404de76c.slice/crio-e77bc9caadafdab7c4e1f0f27f6d89dc8ff7a10381da2b04937b6373a31716aa WatchSource:0}: Error finding container e77bc9caadafdab7c4e1f0f27f6d89dc8ff7a10381da2b04937b6373a31716aa: Status 404 returned error can't find the container with id e77bc9caadafdab7c4e1f0f27f6d89dc8ff7a10381da2b04937b6373a31716aa Nov 02 17:41:07.076106 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:07.073320 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 17:41:07.076106 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:07.073386 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" podUID="9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 17:41:07.526073 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:07.526039 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd" event={"ID":"dfb74ceb-3aea-4cb9-b668-9d92404de76c","Type":"ContainerStarted","Data":"e77bc9caadafdab7c4e1f0f27f6d89dc8ff7a10381da2b04937b6373a31716aa"} Nov 02 17:41:07.920476 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:07.920434 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 17:41:07.920900 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:07.920506 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" podUID="b53fb7b2-d91c-409b-bf83-e24b95a7608f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 17:41:08.072448 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:08.072410 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 17:41:08.072627 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:08.072514 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" podUID="9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 17:41:08.524810 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:08.524725 4860 generic.go:356] "Generic (PLEG): container finished" podID="dfb74ceb-3aea-4cb9-b668-9d92404de76c" containerID="d9429a0acf4d452815f182e653e8d2812dbb89322a1f3e30501bd46c6709d839" exitCode=0 Nov 02 17:41:08.524810 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:08.524802 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd" event={"ID":"dfb74ceb-3aea-4cb9-b668-9d92404de76c","Type":"ContainerDied","Data":"d9429a0acf4d452815f182e653e8d2812dbb89322a1f3e30501bd46c6709d839"} Nov 02 17:41:08.924053 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:08.924021 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:08.924427 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:08.924297 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:08.924934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:08.924918 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 17:41:09.076239 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:09.076210 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:09.076646 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:09.076621 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:09.077411 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:09.077383 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 17:41:09.684304 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:09.684278 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd" Nov 02 17:41:09.736804 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:09.736767 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfb74ceb-3aea-4cb9-b668-9d92404de76c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb74ceb-3aea-4cb9-b668-9d92404de76c-maas-api-openshift-ca\") pod \"dfb74ceb-3aea-4cb9-b668-9d92404de76c\" (UID: \"dfb74ceb-3aea-4cb9-b668-9d92404de76c\") " Nov 02 17:41:09.736993 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:09.736918 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfb74ceb-3aea-4cb9-b668-9d92404de76c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb74ceb-3aea-4cb9-b668-9d92404de76c-maas-api-xks-ca\") pod \"dfb74ceb-3aea-4cb9-b668-9d92404de76c\" (UID: \"dfb74ceb-3aea-4cb9-b668-9d92404de76c\") " Nov 02 17:41:09.737043 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:09.737029 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb74ceb-3aea-4cb9-b668-9d92404de76c-kube-api-access-7j5rr\" (UniqueName: \"kubernetes.io/projected/dfb74ceb-3aea-4cb9-b668-9d92404de76c-kube-api-access-7j5rr\") pod \"dfb74ceb-3aea-4cb9-b668-9d92404de76c\" (UID: \"dfb74ceb-3aea-4cb9-b668-9d92404de76c\") " Nov 02 17:41:09.737203 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:09.737177 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb74ceb-3aea-4cb9-b668-9d92404de76c-maas-api-openshift-ca" pod "dfb74ceb-3aea-4cb9-b668-9d92404de76c" (UID: "dfb74ceb-3aea-4cb9-b668-9d92404de76c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:09.737407 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:09.737391 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb74ceb-3aea-4cb9-b668-9d92404de76c-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:09.739370 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:09.739336 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb74ceb-3aea-4cb9-b668-9d92404de76c-kube-api-access-7j5rr" pod "dfb74ceb-3aea-4cb9-b668-9d92404de76c" (UID: "dfb74ceb-3aea-4cb9-b668-9d92404de76c"). InnerVolumeSpecName "kube-api-access-7j5rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:09.739881 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:09.739820 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfb74ceb-3aea-4cb9-b668-9d92404de76c-maas-api-xks-ca" pod "dfb74ceb-3aea-4cb9-b668-9d92404de76c" (UID: "dfb74ceb-3aea-4cb9-b668-9d92404de76c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:09.838800 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:09.838771 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb74ceb-3aea-4cb9-b668-9d92404de76c-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:09.838800 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:09.838797 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j5rr\" (UniqueName: \"kubernetes.io/projected/dfb74ceb-3aea-4cb9-b668-9d92404de76c-kube-api-access-7j5rr\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:10.526092 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:10.526058 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp"] Nov 02 17:41:10.526653 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:10.526588 4860 cpu_manager.go:414] "containerMap: removing container" podUID="dfb74ceb-3aea-4cb9-b668-9d92404de76c" containerName="revoke-keys" Nov 02 17:41:10.526653 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:10.526599 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb74ceb-3aea-4cb9-b668-9d92404de76c" containerName="revoke-keys" Nov 02 17:41:10.526799 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:10.526702 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb74ceb-3aea-4cb9-b668-9d92404de76c" containerName="revoke-keys" Nov 02 17:41:10.530190 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:10.530169 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp" Nov 02 17:41:10.541083 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:10.541061 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd" Nov 02 17:41:10.541187 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:10.541090 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd" event={"ID":"dfb74ceb-3aea-4cb9-b668-9d92404de76c","Type":"ContainerDied","Data":"e77bc9caadafdab7c4e1f0f27f6d89dc8ff7a10381da2b04937b6373a31716aa"} Nov 02 17:41:10.541187 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:10.541136 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e77bc9caadafdab7c4e1f0f27f6d89dc8ff7a10381da2b04937b6373a31716aa" Nov 02 17:41:10.547962 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:10.546961 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40b19eed-d6d7-4680-b5fc-9ab7e750c851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8705900e-q8dbp\" (UID: \"40b19eed-d6d7-4680-b5fc-9ab7e750c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp" Nov 02 17:41:10.547962 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:10.547011 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40b19eed-d6d7-4680-b5fc-9ab7e750c851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8705900e-q8dbp\" (UID: \"40b19eed-d6d7-4680-b5fc-9ab7e750c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp" Nov 02 17:41:10.547962 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:10.547119 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wxxg\" (UniqueName: \"kubernetes.io/projected/40b19eed-d6d7-4680-b5fc-9ab7e750c851-kube-api-access-7wxxg\") pod \"maas-api-revoke-sub-8705900e-q8dbp\" (UID: \"40b19eed-d6d7-4680-b5fc-9ab7e750c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp" Nov 02 17:41:10.551256 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:10.551234 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp"] Nov 02 17:41:10.648433 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:10.648400 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40b19eed-d6d7-4680-b5fc-9ab7e750c851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8705900e-q8dbp\" (UID: \"40b19eed-d6d7-4680-b5fc-9ab7e750c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp" Nov 02 17:41:10.648651 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:10.648523 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40b19eed-d6d7-4680-b5fc-9ab7e750c851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8705900e-q8dbp\" (UID: \"40b19eed-d6d7-4680-b5fc-9ab7e750c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp" Nov 02 17:41:10.648651 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:10.648638 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wxxg\" (UniqueName: \"kubernetes.io/projected/40b19eed-d6d7-4680-b5fc-9ab7e750c851-kube-api-access-7wxxg\") pod \"maas-api-revoke-sub-8705900e-q8dbp\" (UID: \"40b19eed-d6d7-4680-b5fc-9ab7e750c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp" Nov 02 17:41:10.650123 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:10.650095 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40b19eed-d6d7-4680-b5fc-9ab7e750c851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8705900e-q8dbp\" (UID: \"40b19eed-d6d7-4680-b5fc-9ab7e750c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp" Nov 02 17:41:10.651336 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:10.651308 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40b19eed-d6d7-4680-b5fc-9ab7e750c851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8705900e-q8dbp\" (UID: \"40b19eed-d6d7-4680-b5fc-9ab7e750c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp" Nov 02 17:41:10.672911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:10.672885 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wxxg\" (UniqueName: \"kubernetes.io/projected/40b19eed-d6d7-4680-b5fc-9ab7e750c851-kube-api-access-7wxxg\") pod \"maas-api-revoke-sub-8705900e-q8dbp\" (UID: \"40b19eed-d6d7-4680-b5fc-9ab7e750c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp" Nov 02 17:41:10.847605 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:10.847574 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp" Nov 02 17:41:11.149729 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:41:11.149696 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b19eed_d6d7_4680_b5fc_9ab7e750c851.slice/crio-6e0ecc985166cbed95e60a3a77c6176bff4a715385fa1ea5d48b81f3cf1511e2 WatchSource:0}: Error finding container 6e0ecc985166cbed95e60a3a77c6176bff4a715385fa1ea5d48b81f3cf1511e2: Status 404 returned error can't find the container with id 6e0ecc985166cbed95e60a3a77c6176bff4a715385fa1ea5d48b81f3cf1511e2 Nov 02 17:41:11.150099 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:11.150076 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp"] Nov 02 17:41:11.550311 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:11.550274 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp" event={"ID":"40b19eed-d6d7-4680-b5fc-9ab7e750c851","Type":"ContainerStarted","Data":"de5b9932f440791d80252d51572b5ba5f6f9dc89a1b84aa86a49323658f7d0a1"} Nov 02 17:41:11.550700 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:11.550332 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp" event={"ID":"40b19eed-d6d7-4680-b5fc-9ab7e750c851","Type":"ContainerStarted","Data":"6e0ecc985166cbed95e60a3a77c6176bff4a715385fa1ea5d48b81f3cf1511e2"} Nov 02 17:41:11.601437 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:11.601384 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp" podStartSLOduration=1.6013720500000002 podStartE2EDuration="1.60137205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:11.592366457 +0000 UTC m=+690.213726258" watchObservedRunningTime="2026-11-02 17:41:11.60137205 +0000 UTC m=+690.222731826" Nov 02 17:41:12.558791 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:12.558756 4860 generic.go:356] "Generic (PLEG): container finished" podID="40b19eed-d6d7-4680-b5fc-9ab7e750c851" containerID="de5b9932f440791d80252d51572b5ba5f6f9dc89a1b84aa86a49323658f7d0a1" exitCode=0 Nov 02 17:41:12.559208 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:12.558803 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp" event={"ID":"40b19eed-d6d7-4680-b5fc-9ab7e750c851","Type":"ContainerDied","Data":"de5b9932f440791d80252d51572b5ba5f6f9dc89a1b84aa86a49323658f7d0a1"} Nov 02 17:41:13.372406 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.372372 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq"] Nov 02 17:41:13.376410 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.376390 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq" Nov 02 17:41:13.400640 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.400613 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq"] Nov 02 17:41:13.480344 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.480311 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f32aa8-a2fd-4f77-bca4-671f33875f4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb3a84d1-mt7tq\" (UID: \"73f32aa8-a2fd-4f77-bca4-671f33875f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq" Nov 02 17:41:13.480514 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.480376 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82sxn\" (UniqueName: \"kubernetes.io/projected/73f32aa8-a2fd-4f77-bca4-671f33875f4a-kube-api-access-82sxn\") pod \"maas-api-revoke-sub-bb3a84d1-mt7tq\" (UID: \"73f32aa8-a2fd-4f77-bca4-671f33875f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq" Nov 02 17:41:13.480514 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.480466 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f32aa8-a2fd-4f77-bca4-671f33875f4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb3a84d1-mt7tq\" (UID: \"73f32aa8-a2fd-4f77-bca4-671f33875f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq" Nov 02 17:41:13.582341 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.582304 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f32aa8-a2fd-4f77-bca4-671f33875f4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb3a84d1-mt7tq\" (UID: \"73f32aa8-a2fd-4f77-bca4-671f33875f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq" Nov 02 17:41:13.582829 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.582420 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82sxn\" (UniqueName: \"kubernetes.io/projected/73f32aa8-a2fd-4f77-bca4-671f33875f4a-kube-api-access-82sxn\") pod \"maas-api-revoke-sub-bb3a84d1-mt7tq\" (UID: \"73f32aa8-a2fd-4f77-bca4-671f33875f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq" Nov 02 17:41:13.582829 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.582466 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f32aa8-a2fd-4f77-bca4-671f33875f4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb3a84d1-mt7tq\" (UID: \"73f32aa8-a2fd-4f77-bca4-671f33875f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq" Nov 02 17:41:13.583106 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.583084 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f32aa8-a2fd-4f77-bca4-671f33875f4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb3a84d1-mt7tq\" (UID: \"73f32aa8-a2fd-4f77-bca4-671f33875f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq" Nov 02 17:41:13.584917 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.584893 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f32aa8-a2fd-4f77-bca4-671f33875f4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb3a84d1-mt7tq\" (UID: \"73f32aa8-a2fd-4f77-bca4-671f33875f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq" Nov 02 17:41:13.608181 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.608147 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82sxn\" (UniqueName: \"kubernetes.io/projected/73f32aa8-a2fd-4f77-bca4-671f33875f4a-kube-api-access-82sxn\") pod \"maas-api-revoke-sub-bb3a84d1-mt7tq\" (UID: \"73f32aa8-a2fd-4f77-bca4-671f33875f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq" Nov 02 17:41:13.674241 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.674208 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg"] Nov 02 17:41:13.678789 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.678767 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:13.694075 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.689809 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq" Nov 02 17:41:13.701069 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.701036 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg"] Nov 02 17:41:13.702173 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.701511 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f0a3b8\"/\"distinct-w2-f0a3b8-kserve-self-signed-certs\"" Nov 02 17:41:13.750588 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.750558 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp" Nov 02 17:41:13.786395 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.785384 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40b19eed-d6d7-4680-b5fc-9ab7e750c851-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40b19eed-d6d7-4680-b5fc-9ab7e750c851-maas-api-xks-ca\") pod \"40b19eed-d6d7-4680-b5fc-9ab7e750c851\" (UID: \"40b19eed-d6d7-4680-b5fc-9ab7e750c851\") " Nov 02 17:41:13.786395 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.785447 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b19eed-d6d7-4680-b5fc-9ab7e750c851-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40b19eed-d6d7-4680-b5fc-9ab7e750c851-maas-api-openshift-ca\") pod \"40b19eed-d6d7-4680-b5fc-9ab7e750c851\" (UID: \"40b19eed-d6d7-4680-b5fc-9ab7e750c851\") " Nov 02 17:41:13.786395 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.785491 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b19eed-d6d7-4680-b5fc-9ab7e750c851-kube-api-access-7wxxg\" (UniqueName: \"kubernetes.io/projected/40b19eed-d6d7-4680-b5fc-9ab7e750c851-kube-api-access-7wxxg\") pod \"40b19eed-d6d7-4680-b5fc-9ab7e750c851\" (UID: \"40b19eed-d6d7-4680-b5fc-9ab7e750c851\") " Nov 02 17:41:13.786395 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.785690 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-dshm\") pod \"distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:13.786395 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.785764 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee9ada9-06c5-48ab-b6e3-195c1b445270-tls-certs\") pod \"distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:13.786395 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.785791 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-model-cache\") pod \"distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:13.786395 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.785825 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-tmp-dir\") pod \"distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:13.786395 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.785881 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k8d2\" (UniqueName: \"kubernetes.io/projected/aee9ada9-06c5-48ab-b6e3-195c1b445270-kube-api-access-7k8d2\") pod \"distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:13.786395 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.785940 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-home\") pod \"distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:13.788035 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.787946 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b19eed-d6d7-4680-b5fc-9ab7e750c851-maas-api-openshift-ca" pod "40b19eed-d6d7-4680-b5fc-9ab7e750c851" (UID: "40b19eed-d6d7-4680-b5fc-9ab7e750c851"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:13.789618 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.788961 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b19eed-d6d7-4680-b5fc-9ab7e750c851-maas-api-xks-ca" pod "40b19eed-d6d7-4680-b5fc-9ab7e750c851" (UID: "40b19eed-d6d7-4680-b5fc-9ab7e750c851"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:13.802108 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.791172 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b19eed-d6d7-4680-b5fc-9ab7e750c851-kube-api-access-7wxxg" pod "40b19eed-d6d7-4680-b5fc-9ab7e750c851" (UID: "40b19eed-d6d7-4680-b5fc-9ab7e750c851"). InnerVolumeSpecName "kube-api-access-7wxxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:13.891055 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.890941 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-home\") pod \"distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:13.891235 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.891126 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-dshm\") pod \"distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:13.891235 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.891212 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee9ada9-06c5-48ab-b6e3-195c1b445270-tls-certs\") pod \"distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:13.891361 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.891256 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-model-cache\") pod \"distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:13.891361 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.891288 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-tmp-dir\") pod \"distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:13.891361 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.891338 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k8d2\" (UniqueName: \"kubernetes.io/projected/aee9ada9-06c5-48ab-b6e3-195c1b445270-kube-api-access-7k8d2\") pod \"distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:13.891518 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.891437 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40b19eed-d6d7-4680-b5fc-9ab7e750c851-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:13.891518 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.891454 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40b19eed-d6d7-4680-b5fc-9ab7e750c851-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:13.891518 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.891485 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wxxg\" (UniqueName: \"kubernetes.io/projected/40b19eed-d6d7-4680-b5fc-9ab7e750c851-kube-api-access-7wxxg\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:13.892769 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.892697 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-model-cache\") pod \"distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:13.893052 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.893018 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-tmp-dir\") pod \"distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:13.893765 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.893745 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-home\") pod \"distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:13.895833 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.895788 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-dshm\") pod \"distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:13.897386 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.897079 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee9ada9-06c5-48ab-b6e3-195c1b445270-tls-certs\") pod \"distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:13.916249 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.916219 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k8d2\" (UniqueName: \"kubernetes.io/projected/aee9ada9-06c5-48ab-b6e3-195c1b445270-kube-api-access-7k8d2\") pod \"distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:13.993733 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:13.993700 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:14.008631 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:14.008599 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq"] Nov 02 17:41:14.010197 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:41:14.010160 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f32aa8_a2fd_4f77_bca4_671f33875f4a.slice/crio-53a5d3ca140f8db39369c9acb6b1a2f9cda11fceb6780e70465a3a8dbcf4c4a3 WatchSource:0}: Error finding container 53a5d3ca140f8db39369c9acb6b1a2f9cda11fceb6780e70465a3a8dbcf4c4a3: Status 404 returned error can't find the container with id 53a5d3ca140f8db39369c9acb6b1a2f9cda11fceb6780e70465a3a8dbcf4c4a3 Nov 02 17:41:14.311658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:14.311627 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg"] Nov 02 17:41:14.312698 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:41:14.312674 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaee9ada9_06c5_48ab_b6e3_195c1b445270.slice/crio-fd8ff7de8b53a66ef82c9d7a69795b0cfc976ffe25c45b13da09ad49d54a2e7f WatchSource:0}: Error finding container fd8ff7de8b53a66ef82c9d7a69795b0cfc976ffe25c45b13da09ad49d54a2e7f: Status 404 returned error can't find the container with id fd8ff7de8b53a66ef82c9d7a69795b0cfc976ffe25c45b13da09ad49d54a2e7f Nov 02 17:41:14.579910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:14.579874 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" event={"ID":"aee9ada9-06c5-48ab-b6e3-195c1b445270","Type":"ContainerStarted","Data":"02cba40ea61b2167013c65e8ed83709ca08d0426b48ca2e1a152660cc3bd6b31"} Nov 02 17:41:14.580121 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:14.579924 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" event={"ID":"aee9ada9-06c5-48ab-b6e3-195c1b445270","Type":"ContainerStarted","Data":"fd8ff7de8b53a66ef82c9d7a69795b0cfc976ffe25c45b13da09ad49d54a2e7f"} Nov 02 17:41:14.582007 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:14.581974 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq" event={"ID":"73f32aa8-a2fd-4f77-bca4-671f33875f4a","Type":"ContainerStarted","Data":"42b6bde4e19ef7abfd00ed2d747527278faae48eaec3c14c6bb6a019f54ba979"} Nov 02 17:41:14.582151 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:14.582027 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq" event={"ID":"73f32aa8-a2fd-4f77-bca4-671f33875f4a","Type":"ContainerStarted","Data":"53a5d3ca140f8db39369c9acb6b1a2f9cda11fceb6780e70465a3a8dbcf4c4a3"} Nov 02 17:41:14.583501 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:14.583475 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp" event={"ID":"40b19eed-d6d7-4680-b5fc-9ab7e750c851","Type":"ContainerDied","Data":"6e0ecc985166cbed95e60a3a77c6176bff4a715385fa1ea5d48b81f3cf1511e2"} Nov 02 17:41:14.583811 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:14.583524 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e0ecc985166cbed95e60a3a77c6176bff4a715385fa1ea5d48b81f3cf1511e2" Nov 02 17:41:14.583811 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:14.583483 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp" Nov 02 17:41:14.630980 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:14.630909 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podStartSLOduration=1.630887243 podStartE2EDuration="1.630887243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:14.623703567 +0000 UTC m=+693.245063344" watchObservedRunningTime="2026-11-02 17:41:14.630887243 +0000 UTC m=+693.252247023" Nov 02 17:41:14.693918 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:14.693834 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq" podStartSLOduration=1.69381391 podStartE2EDuration="1.69381391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:14.689370873 +0000 UTC m=+693.310730649" watchObservedRunningTime="2026-11-02 17:41:14.69381391 +0000 UTC m=+693.315173687" Nov 02 17:41:14.994929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:14.994381 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:15.593086 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:15.593049 4860 generic.go:356] "Generic (PLEG): container finished" podID="73f32aa8-a2fd-4f77-bca4-671f33875f4a" containerID="42b6bde4e19ef7abfd00ed2d747527278faae48eaec3c14c6bb6a019f54ba979" exitCode=0 Nov 02 17:41:15.593458 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:15.593130 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq" event={"ID":"73f32aa8-a2fd-4f77-bca4-671f33875f4a","Type":"ContainerDied","Data":"42b6bde4e19ef7abfd00ed2d747527278faae48eaec3c14c6bb6a019f54ba979"} Nov 02 17:41:16.744599 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:16.744571 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq" Nov 02 17:41:16.834128 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:16.834095 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f32aa8-a2fd-4f77-bca4-671f33875f4a-kube-api-access-82sxn\" (UniqueName: \"kubernetes.io/projected/73f32aa8-a2fd-4f77-bca4-671f33875f4a-kube-api-access-82sxn\") pod \"73f32aa8-a2fd-4f77-bca4-671f33875f4a\" (UID: \"73f32aa8-a2fd-4f77-bca4-671f33875f4a\") " Nov 02 17:41:16.834304 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:16.834157 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f32aa8-a2fd-4f77-bca4-671f33875f4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f32aa8-a2fd-4f77-bca4-671f33875f4a-maas-api-openshift-ca\") pod \"73f32aa8-a2fd-4f77-bca4-671f33875f4a\" (UID: \"73f32aa8-a2fd-4f77-bca4-671f33875f4a\") " Nov 02 17:41:16.834353 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:16.834317 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73f32aa8-a2fd-4f77-bca4-671f33875f4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f32aa8-a2fd-4f77-bca4-671f33875f4a-maas-api-xks-ca\") pod \"73f32aa8-a2fd-4f77-bca4-671f33875f4a\" (UID: \"73f32aa8-a2fd-4f77-bca4-671f33875f4a\") " Nov 02 17:41:16.834629 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:16.834597 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f32aa8-a2fd-4f77-bca4-671f33875f4a-maas-api-openshift-ca" pod "73f32aa8-a2fd-4f77-bca4-671f33875f4a" (UID: "73f32aa8-a2fd-4f77-bca4-671f33875f4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:16.834762 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:16.834698 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f32aa8-a2fd-4f77-bca4-671f33875f4a-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:16.836337 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:16.836302 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73f32aa8-a2fd-4f77-bca4-671f33875f4a-maas-api-xks-ca" pod "73f32aa8-a2fd-4f77-bca4-671f33875f4a" (UID: "73f32aa8-a2fd-4f77-bca4-671f33875f4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:16.836432 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:16.836340 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f32aa8-a2fd-4f77-bca4-671f33875f4a-kube-api-access-82sxn" pod "73f32aa8-a2fd-4f77-bca4-671f33875f4a" (UID: "73f32aa8-a2fd-4f77-bca4-671f33875f4a"). InnerVolumeSpecName "kube-api-access-82sxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:16.936112 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:16.936034 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f32aa8-a2fd-4f77-bca4-671f33875f4a-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:16.936112 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:16.936063 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82sxn\" (UniqueName: \"kubernetes.io/projected/73f32aa8-a2fd-4f77-bca4-671f33875f4a-kube-api-access-82sxn\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:17.610252 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:17.610220 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq" Nov 02 17:41:17.610410 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:17.610217 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq" event={"ID":"73f32aa8-a2fd-4f77-bca4-671f33875f4a","Type":"ContainerDied","Data":"53a5d3ca140f8db39369c9acb6b1a2f9cda11fceb6780e70465a3a8dbcf4c4a3"} Nov 02 17:41:17.610410 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:17.610347 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53a5d3ca140f8db39369c9acb6b1a2f9cda11fceb6780e70465a3a8dbcf4c4a3" Nov 02 17:41:17.799144 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:17.799067 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4"] Nov 02 17:41:17.799570 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:17.799557 4860 cpu_manager.go:414] "containerMap: removing container" podUID="73f32aa8-a2fd-4f77-bca4-671f33875f4a" containerName="revoke-keys" Nov 02 17:41:17.799619 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:17.799570 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f32aa8-a2fd-4f77-bca4-671f33875f4a" containerName="revoke-keys" Nov 02 17:41:17.799619 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:17.799594 4860 cpu_manager.go:414] "containerMap: removing container" podUID="40b19eed-d6d7-4680-b5fc-9ab7e750c851" containerName="revoke-keys" Nov 02 17:41:17.799619 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:17.799599 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b19eed-d6d7-4680-b5fc-9ab7e750c851" containerName="revoke-keys" Nov 02 17:41:17.799712 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:17.799703 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f32aa8-a2fd-4f77-bca4-671f33875f4a" containerName="revoke-keys" Nov 02 17:41:17.799748 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:17.799718 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b19eed-d6d7-4680-b5fc-9ab7e750c851" containerName="revoke-keys" Nov 02 17:41:17.803336 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:17.803319 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4" Nov 02 17:41:17.823764 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:17.823734 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4"] Nov 02 17:41:17.827613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:17.827585 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:41:17.827770 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:17.827619 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:41:17.946359 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:17.946326 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nspbg\" (UniqueName: \"kubernetes.io/projected/0cad1749-01a6-48b5-91b2-55b83107716c-kube-api-access-nspbg\") pod \"maas-api-revoke-sub-1958d0f3-gxpt4\" (UID: \"0cad1749-01a6-48b5-91b2-55b83107716c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4" Nov 02 17:41:17.946534 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:17.946367 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cad1749-01a6-48b5-91b2-55b83107716c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1958d0f3-gxpt4\" (UID: \"0cad1749-01a6-48b5-91b2-55b83107716c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4" Nov 02 17:41:17.946534 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:17.946500 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cad1749-01a6-48b5-91b2-55b83107716c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1958d0f3-gxpt4\" (UID: \"0cad1749-01a6-48b5-91b2-55b83107716c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4" Nov 02 17:41:18.048081 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:18.048048 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cad1749-01a6-48b5-91b2-55b83107716c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1958d0f3-gxpt4\" (UID: \"0cad1749-01a6-48b5-91b2-55b83107716c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4" Nov 02 17:41:18.048278 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:18.048210 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nspbg\" (UniqueName: \"kubernetes.io/projected/0cad1749-01a6-48b5-91b2-55b83107716c-kube-api-access-nspbg\") pod \"maas-api-revoke-sub-1958d0f3-gxpt4\" (UID: \"0cad1749-01a6-48b5-91b2-55b83107716c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4" Nov 02 17:41:18.048278 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:18.048249 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cad1749-01a6-48b5-91b2-55b83107716c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1958d0f3-gxpt4\" (UID: \"0cad1749-01a6-48b5-91b2-55b83107716c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4" Nov 02 17:41:18.048949 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:18.048931 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cad1749-01a6-48b5-91b2-55b83107716c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1958d0f3-gxpt4\" (UID: \"0cad1749-01a6-48b5-91b2-55b83107716c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4" Nov 02 17:41:18.050728 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:18.050678 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cad1749-01a6-48b5-91b2-55b83107716c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1958d0f3-gxpt4\" (UID: \"0cad1749-01a6-48b5-91b2-55b83107716c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4" Nov 02 17:41:18.072486 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:18.072454 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nspbg\" (UniqueName: \"kubernetes.io/projected/0cad1749-01a6-48b5-91b2-55b83107716c-kube-api-access-nspbg\") pod \"maas-api-revoke-sub-1958d0f3-gxpt4\" (UID: \"0cad1749-01a6-48b5-91b2-55b83107716c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4" Nov 02 17:41:18.118029 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:18.117995 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4" Nov 02 17:41:18.425367 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:18.425338 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4"] Nov 02 17:41:18.426100 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:41:18.426078 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cad1749_01a6_48b5_91b2_55b83107716c.slice/crio-d794438bf0c4d2ae4fd5cba30aa0618549de4cfd59773a1e5c83b435031baf53 WatchSource:0}: Error finding container d794438bf0c4d2ae4fd5cba30aa0618549de4cfd59773a1e5c83b435031baf53: Status 404 returned error can't find the container with id d794438bf0c4d2ae4fd5cba30aa0618549de4cfd59773a1e5c83b435031baf53 Nov 02 17:41:18.618789 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:18.618754 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4" event={"ID":"0cad1749-01a6-48b5-91b2-55b83107716c","Type":"ContainerStarted","Data":"d794438bf0c4d2ae4fd5cba30aa0618549de4cfd59773a1e5c83b435031baf53"} Nov 02 17:41:19.627632 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:19.627597 4860 generic.go:356] "Generic (PLEG): container finished" podID="0cad1749-01a6-48b5-91b2-55b83107716c" containerID="88b6743b6a7c6771235d3186ec334b2285eb8ae01f1aa6ec078166140659517c" exitCode=0 Nov 02 17:41:19.628064 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:19.627677 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4" event={"ID":"0cad1749-01a6-48b5-91b2-55b83107716c","Type":"ContainerDied","Data":"88b6743b6a7c6771235d3186ec334b2285eb8ae01f1aa6ec078166140659517c"} Nov 02 17:41:20.784064 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:20.784039 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4" Nov 02 17:41:20.876963 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:20.876917 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cad1749-01a6-48b5-91b2-55b83107716c-kube-api-access-nspbg\" (UniqueName: \"kubernetes.io/projected/0cad1749-01a6-48b5-91b2-55b83107716c-kube-api-access-nspbg\") pod \"0cad1749-01a6-48b5-91b2-55b83107716c\" (UID: \"0cad1749-01a6-48b5-91b2-55b83107716c\") " Nov 02 17:41:20.876963 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:20.876968 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cad1749-01a6-48b5-91b2-55b83107716c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cad1749-01a6-48b5-91b2-55b83107716c-maas-api-xks-ca\") pod \"0cad1749-01a6-48b5-91b2-55b83107716c\" (UID: \"0cad1749-01a6-48b5-91b2-55b83107716c\") " Nov 02 17:41:20.877219 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:20.877046 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cad1749-01a6-48b5-91b2-55b83107716c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cad1749-01a6-48b5-91b2-55b83107716c-maas-api-openshift-ca\") pod \"0cad1749-01a6-48b5-91b2-55b83107716c\" (UID: \"0cad1749-01a6-48b5-91b2-55b83107716c\") " Nov 02 17:41:20.877446 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:20.877409 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cad1749-01a6-48b5-91b2-55b83107716c-maas-api-openshift-ca" pod "0cad1749-01a6-48b5-91b2-55b83107716c" (UID: "0cad1749-01a6-48b5-91b2-55b83107716c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:20.879144 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:20.879122 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cad1749-01a6-48b5-91b2-55b83107716c-maas-api-xks-ca" pod "0cad1749-01a6-48b5-91b2-55b83107716c" (UID: "0cad1749-01a6-48b5-91b2-55b83107716c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:20.879217 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:20.879182 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cad1749-01a6-48b5-91b2-55b83107716c-kube-api-access-nspbg" pod "0cad1749-01a6-48b5-91b2-55b83107716c" (UID: "0cad1749-01a6-48b5-91b2-55b83107716c"). InnerVolumeSpecName "kube-api-access-nspbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:20.978717 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:20.978683 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nspbg\" (UniqueName: \"kubernetes.io/projected/0cad1749-01a6-48b5-91b2-55b83107716c-kube-api-access-nspbg\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:20.978717 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:20.978712 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cad1749-01a6-48b5-91b2-55b83107716c-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:20.978717 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:20.978724 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cad1749-01a6-48b5-91b2-55b83107716c-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:21.642900 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:21.642837 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4" Nov 02 17:41:21.643152 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:21.642875 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4" event={"ID":"0cad1749-01a6-48b5-91b2-55b83107716c","Type":"ContainerDied","Data":"d794438bf0c4d2ae4fd5cba30aa0618549de4cfd59773a1e5c83b435031baf53"} Nov 02 17:41:21.643152 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:21.642989 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d794438bf0c4d2ae4fd5cba30aa0618549de4cfd59773a1e5c83b435031baf53" Nov 02 17:41:21.770830 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:41:21.770722 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:41:22.213385 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:22.213351 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v"] Nov 02 17:41:22.213905 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:22.213891 4860 cpu_manager.go:414] "containerMap: removing container" podUID="0cad1749-01a6-48b5-91b2-55b83107716c" containerName="revoke-keys" Nov 02 17:41:22.213905 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:22.213905 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cad1749-01a6-48b5-91b2-55b83107716c" containerName="revoke-keys" Nov 02 17:41:22.214040 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:22.214024 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cad1749-01a6-48b5-91b2-55b83107716c" containerName="revoke-keys" Nov 02 17:41:22.217550 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:22.217527 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v" Nov 02 17:41:22.238120 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:22.238077 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v"] Nov 02 17:41:22.244678 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:22.244650 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:41:22.244678 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:22.244659 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:41:22.291786 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:22.291755 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzchz\" (UniqueName: \"kubernetes.io/projected/63da40ed-b7f0-458c-9e88-aa8dae1461a5-kube-api-access-wzchz\") pod \"maas-api-revoke-sub-24b018f2-h4t4v\" (UID: \"63da40ed-b7f0-458c-9e88-aa8dae1461a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v" Nov 02 17:41:22.291986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:22.291946 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63da40ed-b7f0-458c-9e88-aa8dae1461a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24b018f2-h4t4v\" (UID: \"63da40ed-b7f0-458c-9e88-aa8dae1461a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v" Nov 02 17:41:22.292054 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:22.291995 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63da40ed-b7f0-458c-9e88-aa8dae1461a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24b018f2-h4t4v\" (UID: \"63da40ed-b7f0-458c-9e88-aa8dae1461a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v" Nov 02 17:41:22.393063 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:22.393028 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63da40ed-b7f0-458c-9e88-aa8dae1461a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24b018f2-h4t4v\" (UID: \"63da40ed-b7f0-458c-9e88-aa8dae1461a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v" Nov 02 17:41:22.393226 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:22.393077 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63da40ed-b7f0-458c-9e88-aa8dae1461a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24b018f2-h4t4v\" (UID: \"63da40ed-b7f0-458c-9e88-aa8dae1461a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v" Nov 02 17:41:22.393226 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:22.393131 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzchz\" (UniqueName: \"kubernetes.io/projected/63da40ed-b7f0-458c-9e88-aa8dae1461a5-kube-api-access-wzchz\") pod \"maas-api-revoke-sub-24b018f2-h4t4v\" (UID: \"63da40ed-b7f0-458c-9e88-aa8dae1461a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v" Nov 02 17:41:22.393844 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:22.393823 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63da40ed-b7f0-458c-9e88-aa8dae1461a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24b018f2-h4t4v\" (UID: \"63da40ed-b7f0-458c-9e88-aa8dae1461a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v" Nov 02 17:41:22.395511 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:22.395488 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63da40ed-b7f0-458c-9e88-aa8dae1461a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24b018f2-h4t4v\" (UID: \"63da40ed-b7f0-458c-9e88-aa8dae1461a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v" Nov 02 17:41:22.416629 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:22.416605 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzchz\" (UniqueName: \"kubernetes.io/projected/63da40ed-b7f0-458c-9e88-aa8dae1461a5-kube-api-access-wzchz\") pod \"maas-api-revoke-sub-24b018f2-h4t4v\" (UID: \"63da40ed-b7f0-458c-9e88-aa8dae1461a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v" Nov 02 17:41:22.531237 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:22.531161 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v" Nov 02 17:41:22.833679 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:41:22.833645 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63da40ed_b7f0_458c_9e88_aa8dae1461a5.slice/crio-217eb9445df9fcef8553f59861e573873fe56a7ad0f994ad4b437e90c125be17 WatchSource:0}: Error finding container 217eb9445df9fcef8553f59861e573873fe56a7ad0f994ad4b437e90c125be17: Status 404 returned error can't find the container with id 217eb9445df9fcef8553f59861e573873fe56a7ad0f994ad4b437e90c125be17 Nov 02 17:41:22.833841 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:22.833821 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v"] Nov 02 17:41:23.661284 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:23.661246 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v" event={"ID":"63da40ed-b7f0-458c-9e88-aa8dae1461a5","Type":"ContainerStarted","Data":"38c43e0230b35c0ff04def00c8783931e373341d30086a43ed80bba13077c5c6"} Nov 02 17:41:23.661755 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:23.661317 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v" event={"ID":"63da40ed-b7f0-458c-9e88-aa8dae1461a5","Type":"ContainerStarted","Data":"217eb9445df9fcef8553f59861e573873fe56a7ad0f994ad4b437e90c125be17"} Nov 02 17:41:23.702648 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:23.702590 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v" podStartSLOduration=1.7025764159999999 podStartE2EDuration="1.702576416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:23.701083954 +0000 UTC m=+702.322443729" watchObservedRunningTime="2026-11-02 17:41:23.702576416 +0000 UTC m=+702.323936224" Nov 02 17:41:23.994032 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:23.993953 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:24.007154 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:24.007124 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:24.545730 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:24.545686 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj"] Nov 02 17:41:24.550969 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:24.550948 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj" Nov 02 17:41:24.570947 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:24.570915 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj"] Nov 02 17:41:24.669942 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:24.669904 4860 generic.go:356] "Generic (PLEG): container finished" podID="63da40ed-b7f0-458c-9e88-aa8dae1461a5" containerID="38c43e0230b35c0ff04def00c8783931e373341d30086a43ed80bba13077c5c6" exitCode=0 Nov 02 17:41:24.670420 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:24.669998 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v" event={"ID":"63da40ed-b7f0-458c-9e88-aa8dae1461a5","Type":"ContainerDied","Data":"38c43e0230b35c0ff04def00c8783931e373341d30086a43ed80bba13077c5c6"} Nov 02 17:41:24.682028 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:24.682000 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:41:24.720376 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:24.720341 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x52qr\" (UniqueName: \"kubernetes.io/projected/cff84bf9-4916-469a-ba35-048a68ae6c92-kube-api-access-x52qr\") pod \"maas-api-revoke-sub-12488095-87xgj\" (UID: \"cff84bf9-4916-469a-ba35-048a68ae6c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj" Nov 02 17:41:24.720541 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:24.720405 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff84bf9-4916-469a-ba35-048a68ae6c92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12488095-87xgj\" (UID: \"cff84bf9-4916-469a-ba35-048a68ae6c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj" Nov 02 17:41:24.720592 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:24.720571 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff84bf9-4916-469a-ba35-048a68ae6c92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12488095-87xgj\" (UID: \"cff84bf9-4916-469a-ba35-048a68ae6c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj" Nov 02 17:41:24.821689 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:24.821653 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x52qr\" (UniqueName: \"kubernetes.io/projected/cff84bf9-4916-469a-ba35-048a68ae6c92-kube-api-access-x52qr\") pod \"maas-api-revoke-sub-12488095-87xgj\" (UID: \"cff84bf9-4916-469a-ba35-048a68ae6c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj" Nov 02 17:41:24.821895 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:24.821701 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff84bf9-4916-469a-ba35-048a68ae6c92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12488095-87xgj\" (UID: \"cff84bf9-4916-469a-ba35-048a68ae6c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj" Nov 02 17:41:24.821895 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:24.821735 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff84bf9-4916-469a-ba35-048a68ae6c92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12488095-87xgj\" (UID: \"cff84bf9-4916-469a-ba35-048a68ae6c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj" Nov 02 17:41:24.822377 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:24.822359 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff84bf9-4916-469a-ba35-048a68ae6c92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12488095-87xgj\" (UID: \"cff84bf9-4916-469a-ba35-048a68ae6c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj" Nov 02 17:41:24.824081 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:24.824061 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff84bf9-4916-469a-ba35-048a68ae6c92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12488095-87xgj\" (UID: \"cff84bf9-4916-469a-ba35-048a68ae6c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj" Nov 02 17:41:24.860807 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:24.860778 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x52qr\" (UniqueName: \"kubernetes.io/projected/cff84bf9-4916-469a-ba35-048a68ae6c92-kube-api-access-x52qr\") pod \"maas-api-revoke-sub-12488095-87xgj\" (UID: \"cff84bf9-4916-469a-ba35-048a68ae6c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj" Nov 02 17:41:24.864606 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:24.864584 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj" Nov 02 17:41:25.171729 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:25.171698 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj"] Nov 02 17:41:25.172921 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:41:25.172889 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff84bf9_4916_469a_ba35_048a68ae6c92.slice/crio-4d1e62854eb5e31fe0094c8e2ed2391090191d3e7aed67ab16a4549325ceec52 WatchSource:0}: Error finding container 4d1e62854eb5e31fe0094c8e2ed2391090191d3e7aed67ab16a4549325ceec52: Status 404 returned error can't find the container with id 4d1e62854eb5e31fe0094c8e2ed2391090191d3e7aed67ab16a4549325ceec52 Nov 02 17:41:25.680677 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:25.680637 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj" event={"ID":"cff84bf9-4916-469a-ba35-048a68ae6c92","Type":"ContainerStarted","Data":"09df0d7a6b0663308be7f8f0cefc3e9c1587d14c152a11b4c72b9c4aff7a242b"} Nov 02 17:41:25.681143 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:25.680704 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj" event={"ID":"cff84bf9-4916-469a-ba35-048a68ae6c92","Type":"ContainerStarted","Data":"4d1e62854eb5e31fe0094c8e2ed2391090191d3e7aed67ab16a4549325ceec52"} Nov 02 17:41:25.725403 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:25.725300 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj" podStartSLOduration=1.725286263 podStartE2EDuration="1.725286263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:25.723699158 +0000 UTC m=+704.345058933" watchObservedRunningTime="2026-11-02 17:41:25.725286263 +0000 UTC m=+704.346646066" Nov 02 17:41:25.819458 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:25.819431 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v" Nov 02 17:41:25.935326 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:25.935288 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63da40ed-b7f0-458c-9e88-aa8dae1461a5-kube-api-access-wzchz\" (UniqueName: \"kubernetes.io/projected/63da40ed-b7f0-458c-9e88-aa8dae1461a5-kube-api-access-wzchz\") pod \"63da40ed-b7f0-458c-9e88-aa8dae1461a5\" (UID: \"63da40ed-b7f0-458c-9e88-aa8dae1461a5\") " Nov 02 17:41:25.935326 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:25.935332 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63da40ed-b7f0-458c-9e88-aa8dae1461a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63da40ed-b7f0-458c-9e88-aa8dae1461a5-maas-api-openshift-ca\") pod \"63da40ed-b7f0-458c-9e88-aa8dae1461a5\" (UID: \"63da40ed-b7f0-458c-9e88-aa8dae1461a5\") " Nov 02 17:41:25.935525 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:25.935429 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63da40ed-b7f0-458c-9e88-aa8dae1461a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63da40ed-b7f0-458c-9e88-aa8dae1461a5-maas-api-xks-ca\") pod \"63da40ed-b7f0-458c-9e88-aa8dae1461a5\" (UID: \"63da40ed-b7f0-458c-9e88-aa8dae1461a5\") " Nov 02 17:41:25.935813 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:25.935786 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63da40ed-b7f0-458c-9e88-aa8dae1461a5-maas-api-openshift-ca" pod "63da40ed-b7f0-458c-9e88-aa8dae1461a5" (UID: "63da40ed-b7f0-458c-9e88-aa8dae1461a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:25.937411 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:25.937387 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63da40ed-b7f0-458c-9e88-aa8dae1461a5-kube-api-access-wzchz" pod "63da40ed-b7f0-458c-9e88-aa8dae1461a5" (UID: "63da40ed-b7f0-458c-9e88-aa8dae1461a5"). InnerVolumeSpecName "kube-api-access-wzchz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:25.937627 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:25.937615 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63da40ed-b7f0-458c-9e88-aa8dae1461a5-maas-api-xks-ca" pod "63da40ed-b7f0-458c-9e88-aa8dae1461a5" (UID: "63da40ed-b7f0-458c-9e88-aa8dae1461a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:26.036936 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:26.036838 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzchz\" (UniqueName: \"kubernetes.io/projected/63da40ed-b7f0-458c-9e88-aa8dae1461a5-kube-api-access-wzchz\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:26.036936 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:26.036884 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63da40ed-b7f0-458c-9e88-aa8dae1461a5-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:26.036936 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:26.036895 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63da40ed-b7f0-458c-9e88-aa8dae1461a5-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:26.690389 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:26.690364 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v" Nov 02 17:41:26.690871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:26.690398 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v" event={"ID":"63da40ed-b7f0-458c-9e88-aa8dae1461a5","Type":"ContainerDied","Data":"217eb9445df9fcef8553f59861e573873fe56a7ad0f994ad4b437e90c125be17"} Nov 02 17:41:26.690871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:26.690453 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="217eb9445df9fcef8553f59861e573873fe56a7ad0f994ad4b437e90c125be17" Nov 02 17:41:26.692104 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:26.692075 4860 generic.go:356] "Generic (PLEG): container finished" podID="cff84bf9-4916-469a-ba35-048a68ae6c92" containerID="09df0d7a6b0663308be7f8f0cefc3e9c1587d14c152a11b4c72b9c4aff7a242b" exitCode=0 Nov 02 17:41:26.692232 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:26.692147 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj" event={"ID":"cff84bf9-4916-469a-ba35-048a68ae6c92","Type":"ContainerDied","Data":"09df0d7a6b0663308be7f8f0cefc3e9c1587d14c152a11b4c72b9c4aff7a242b"} Nov 02 17:41:27.873739 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:27.873714 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj" Nov 02 17:41:27.957471 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:27.957392 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cff84bf9-4916-469a-ba35-048a68ae6c92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff84bf9-4916-469a-ba35-048a68ae6c92-maas-api-openshift-ca\") pod \"cff84bf9-4916-469a-ba35-048a68ae6c92\" (UID: \"cff84bf9-4916-469a-ba35-048a68ae6c92\") " Nov 02 17:41:27.957471 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:27.957454 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cff84bf9-4916-469a-ba35-048a68ae6c92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff84bf9-4916-469a-ba35-048a68ae6c92-maas-api-xks-ca\") pod \"cff84bf9-4916-469a-ba35-048a68ae6c92\" (UID: \"cff84bf9-4916-469a-ba35-048a68ae6c92\") " Nov 02 17:41:27.957664 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:27.957504 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff84bf9-4916-469a-ba35-048a68ae6c92-kube-api-access-x52qr\" (UniqueName: \"kubernetes.io/projected/cff84bf9-4916-469a-ba35-048a68ae6c92-kube-api-access-x52qr\") pod \"cff84bf9-4916-469a-ba35-048a68ae6c92\" (UID: \"cff84bf9-4916-469a-ba35-048a68ae6c92\") " Nov 02 17:41:27.957773 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:27.957743 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff84bf9-4916-469a-ba35-048a68ae6c92-maas-api-openshift-ca" pod "cff84bf9-4916-469a-ba35-048a68ae6c92" (UID: "cff84bf9-4916-469a-ba35-048a68ae6c92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:27.957964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:27.957941 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff84bf9-4916-469a-ba35-048a68ae6c92-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:27.959579 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:27.959555 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff84bf9-4916-469a-ba35-048a68ae6c92-kube-api-access-x52qr" pod "cff84bf9-4916-469a-ba35-048a68ae6c92" (UID: "cff84bf9-4916-469a-ba35-048a68ae6c92"). InnerVolumeSpecName "kube-api-access-x52qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:27.959579 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:27.959558 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cff84bf9-4916-469a-ba35-048a68ae6c92-maas-api-xks-ca" pod "cff84bf9-4916-469a-ba35-048a68ae6c92" (UID: "cff84bf9-4916-469a-ba35-048a68ae6c92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:28.059470 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:28.059436 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff84bf9-4916-469a-ba35-048a68ae6c92-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:28.059470 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:28.059469 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x52qr\" (UniqueName: \"kubernetes.io/projected/cff84bf9-4916-469a-ba35-048a68ae6c92-kube-api-access-x52qr\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:28.707357 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:28.707322 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj" Nov 02 17:41:28.707540 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:28.707323 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj" event={"ID":"cff84bf9-4916-469a-ba35-048a68ae6c92","Type":"ContainerDied","Data":"4d1e62854eb5e31fe0094c8e2ed2391090191d3e7aed67ab16a4549325ceec52"} Nov 02 17:41:28.707540 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:28.707448 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d1e62854eb5e31fe0094c8e2ed2391090191d3e7aed67ab16a4549325ceec52" Nov 02 17:41:30.547873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.546209 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8"] Nov 02 17:41:30.547873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.547039 4860 cpu_manager.go:414] "containerMap: removing container" podUID="cff84bf9-4916-469a-ba35-048a68ae6c92" containerName="revoke-keys" Nov 02 17:41:30.547873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.547059 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff84bf9-4916-469a-ba35-048a68ae6c92" containerName="revoke-keys" Nov 02 17:41:30.547873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.547107 4860 cpu_manager.go:414] "containerMap: removing container" podUID="63da40ed-b7f0-458c-9e88-aa8dae1461a5" containerName="revoke-keys" Nov 02 17:41:30.547873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.547116 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63da40ed-b7f0-458c-9e88-aa8dae1461a5" containerName="revoke-keys" Nov 02 17:41:30.547873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.547256 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="63da40ed-b7f0-458c-9e88-aa8dae1461a5" containerName="revoke-keys" Nov 02 17:41:30.547873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.547274 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff84bf9-4916-469a-ba35-048a68ae6c92" containerName="revoke-keys" Nov 02 17:41:30.552869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.551719 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8" Nov 02 17:41:30.578242 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.578207 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8"] Nov 02 17:41:30.578572 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.578555 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:41:30.578641 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.578572 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:41:30.690282 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.690239 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf4ab94-d6ad-460b-a8bb-681833072410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19497ce5-j7ln8\" (UID: \"edf4ab94-d6ad-460b-a8bb-681833072410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8" Nov 02 17:41:30.690496 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.690357 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf4ab94-d6ad-460b-a8bb-681833072410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19497ce5-j7ln8\" (UID: \"edf4ab94-d6ad-460b-a8bb-681833072410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8" Nov 02 17:41:30.690496 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.690393 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzstp\" (UniqueName: \"kubernetes.io/projected/edf4ab94-d6ad-460b-a8bb-681833072410-kube-api-access-bzstp\") pod \"maas-api-revoke-sub-19497ce5-j7ln8\" (UID: \"edf4ab94-d6ad-460b-a8bb-681833072410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8" Nov 02 17:41:30.791396 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.791356 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf4ab94-d6ad-460b-a8bb-681833072410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19497ce5-j7ln8\" (UID: \"edf4ab94-d6ad-460b-a8bb-681833072410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8" Nov 02 17:41:30.791603 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.791511 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf4ab94-d6ad-460b-a8bb-681833072410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19497ce5-j7ln8\" (UID: \"edf4ab94-d6ad-460b-a8bb-681833072410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8" Nov 02 17:41:30.791603 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.791543 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzstp\" (UniqueName: \"kubernetes.io/projected/edf4ab94-d6ad-460b-a8bb-681833072410-kube-api-access-bzstp\") pod \"maas-api-revoke-sub-19497ce5-j7ln8\" (UID: \"edf4ab94-d6ad-460b-a8bb-681833072410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8" Nov 02 17:41:30.792121 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.792085 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf4ab94-d6ad-460b-a8bb-681833072410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19497ce5-j7ln8\" (UID: \"edf4ab94-d6ad-460b-a8bb-681833072410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8" Nov 02 17:41:30.794166 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.794137 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf4ab94-d6ad-460b-a8bb-681833072410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19497ce5-j7ln8\" (UID: \"edf4ab94-d6ad-460b-a8bb-681833072410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8" Nov 02 17:41:30.816086 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.816053 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzstp\" (UniqueName: \"kubernetes.io/projected/edf4ab94-d6ad-460b-a8bb-681833072410-kube-api-access-bzstp\") pod \"maas-api-revoke-sub-19497ce5-j7ln8\" (UID: \"edf4ab94-d6ad-460b-a8bb-681833072410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8" Nov 02 17:41:30.881859 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:30.881817 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8" Nov 02 17:41:31.308320 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:31.308272 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8"] Nov 02 17:41:31.736080 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:31.736041 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8" event={"ID":"edf4ab94-d6ad-460b-a8bb-681833072410","Type":"ContainerStarted","Data":"86c1e5292175283cb9e2f90496a67606231ecfee1a77a590a1d6a16d25457433"} Nov 02 17:41:31.736472 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:31.736103 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8" event={"ID":"edf4ab94-d6ad-460b-a8bb-681833072410","Type":"ContainerStarted","Data":"6eb03a30984c42c5fa8f66a01ba5e1393543a2ac464c3aba637d75eea0e998c4"} Nov 02 17:41:31.780419 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:31.780370 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8" podStartSLOduration=1.780358278 podStartE2EDuration="1.780358278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:31.779239872 +0000 UTC m=+710.400599647" watchObservedRunningTime="2026-11-02 17:41:31.780358278 +0000 UTC m=+710.401718053" Nov 02 17:41:32.744433 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:32.744398 4860 generic.go:356] "Generic (PLEG): container finished" podID="edf4ab94-d6ad-460b-a8bb-681833072410" containerID="86c1e5292175283cb9e2f90496a67606231ecfee1a77a590a1d6a16d25457433" exitCode=0 Nov 02 17:41:32.744801 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:32.744476 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8" event={"ID":"edf4ab94-d6ad-460b-a8bb-681833072410","Type":"ContainerDied","Data":"86c1e5292175283cb9e2f90496a67606231ecfee1a77a590a1d6a16d25457433"} Nov 02 17:41:33.226449 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.226415 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg"] Nov 02 17:41:33.232042 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.232006 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.251739 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.251707 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg"] Nov 02 17:41:33.258199 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.258175 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2899ab\"/\"distinct-w4-2899ab-kserve-self-signed-certs\"" Nov 02 17:41:33.340266 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.340231 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-tls-certs\") pod \"distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.340266 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.340277 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-dshm\") pod \"distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.340531 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.340467 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmcvs\" (UniqueName: \"kubernetes.io/projected/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-kube-api-access-fmcvs\") pod \"distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.340592 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.340558 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-tmp-dir\") pod \"distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.340699 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.340609 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-home\") pod \"distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.340699 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.340668 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-model-cache\") pod \"distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.441784 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.441750 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmcvs\" (UniqueName: \"kubernetes.io/projected/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-kube-api-access-fmcvs\") pod \"distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.441953 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.441817 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-tmp-dir\") pod \"distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.441953 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.441924 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-home\") pod \"distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.442055 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.441969 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-model-cache\") pod \"distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.442055 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.442033 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-tls-certs\") pod \"distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.442173 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.442068 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-dshm\") pod \"distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.442359 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.442332 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-tmp-dir\") pod \"distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.442632 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.442602 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-model-cache\") pod \"distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.442716 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.442646 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-home\") pod \"distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.444665 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.444620 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-tls-certs\") pod \"distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.444777 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.444683 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-dshm\") pod \"distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.468185 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.468141 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmcvs\" (UniqueName: \"kubernetes.io/projected/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-kube-api-access-fmcvs\") pod \"distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.546632 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.546321 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:33.859505 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.859469 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg"] Nov 02 17:41:33.864702 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:41:33.863017 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d64cc17_abaf_4e5e_bb0d_b2ed1338437c.slice/crio-e6ecd99f57c5ce2a83c72eceb421ecd76f9f78b4d6709e96c91ab06ff33bcbf5 WatchSource:0}: Error finding container e6ecd99f57c5ce2a83c72eceb421ecd76f9f78b4d6709e96c91ab06ff33bcbf5: Status 404 returned error can't find the container with id e6ecd99f57c5ce2a83c72eceb421ecd76f9f78b4d6709e96c91ab06ff33bcbf5 Nov 02 17:41:33.966450 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:33.966424 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8" Nov 02 17:41:34.047913 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:34.047879 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf4ab94-d6ad-460b-a8bb-681833072410-kube-api-access-bzstp\" (UniqueName: \"kubernetes.io/projected/edf4ab94-d6ad-460b-a8bb-681833072410-kube-api-access-bzstp\") pod \"edf4ab94-d6ad-460b-a8bb-681833072410\" (UID: \"edf4ab94-d6ad-460b-a8bb-681833072410\") " Nov 02 17:41:34.048079 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:34.047978 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edf4ab94-d6ad-460b-a8bb-681833072410-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf4ab94-d6ad-460b-a8bb-681833072410-maas-api-openshift-ca\") pod \"edf4ab94-d6ad-460b-a8bb-681833072410\" (UID: \"edf4ab94-d6ad-460b-a8bb-681833072410\") " Nov 02 17:41:34.048079 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:34.048051 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edf4ab94-d6ad-460b-a8bb-681833072410-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf4ab94-d6ad-460b-a8bb-681833072410-maas-api-xks-ca\") pod \"edf4ab94-d6ad-460b-a8bb-681833072410\" (UID: \"edf4ab94-d6ad-460b-a8bb-681833072410\") " Nov 02 17:41:34.048452 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:34.048327 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf4ab94-d6ad-460b-a8bb-681833072410-maas-api-openshift-ca" pod "edf4ab94-d6ad-460b-a8bb-681833072410" (UID: "edf4ab94-d6ad-460b-a8bb-681833072410"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:34.049963 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:34.049939 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf4ab94-d6ad-460b-a8bb-681833072410-kube-api-access-bzstp" pod "edf4ab94-d6ad-460b-a8bb-681833072410" (UID: "edf4ab94-d6ad-460b-a8bb-681833072410"). InnerVolumeSpecName "kube-api-access-bzstp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:34.050083 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:34.050066 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf4ab94-d6ad-460b-a8bb-681833072410-maas-api-xks-ca" pod "edf4ab94-d6ad-460b-a8bb-681833072410" (UID: "edf4ab94-d6ad-460b-a8bb-681833072410"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:34.149362 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:34.149267 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf4ab94-d6ad-460b-a8bb-681833072410-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:34.149362 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:34.149301 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf4ab94-d6ad-460b-a8bb-681833072410-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:34.149362 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:34.149312 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzstp\" (UniqueName: \"kubernetes.io/projected/edf4ab94-d6ad-460b-a8bb-681833072410-kube-api-access-bzstp\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:34.766163 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:34.766056 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8" Nov 02 17:41:34.766163 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:34.766088 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8" event={"ID":"edf4ab94-d6ad-460b-a8bb-681833072410","Type":"ContainerDied","Data":"6eb03a30984c42c5fa8f66a01ba5e1393543a2ac464c3aba637d75eea0e998c4"} Nov 02 17:41:34.766163 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:34.766151 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eb03a30984c42c5fa8f66a01ba5e1393543a2ac464c3aba637d75eea0e998c4" Nov 02 17:41:34.768259 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:34.768230 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" event={"ID":"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c","Type":"ContainerStarted","Data":"37073502e6da2fd11871626b4293fca85bf9b77558882532ccef5c0702c1b573"} Nov 02 17:41:34.768401 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:34.768274 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" event={"ID":"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c","Type":"ContainerStarted","Data":"e6ecd99f57c5ce2a83c72eceb421ecd76f9f78b4d6709e96c91ab06ff33bcbf5"} Nov 02 17:41:34.812634 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:34.812580 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podStartSLOduration=1.812566221 podStartE2EDuration="1.812566221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:34.810167228 +0000 UTC m=+713.431527005" watchObservedRunningTime="2026-11-02 17:41:34.812566221 +0000 UTC m=+713.433925996" Nov 02 17:41:35.546915 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:35.546881 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:38.766969 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:38.766936 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n"] Nov 02 17:41:38.767496 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:38.767481 4860 cpu_manager.go:414] "containerMap: removing container" podUID="edf4ab94-d6ad-460b-a8bb-681833072410" containerName="revoke-keys" Nov 02 17:41:38.767496 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:38.767496 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf4ab94-d6ad-460b-a8bb-681833072410" containerName="revoke-keys" Nov 02 17:41:38.767604 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:38.767589 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="edf4ab94-d6ad-460b-a8bb-681833072410" containerName="revoke-keys" Nov 02 17:41:38.780240 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:38.780213 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n" Nov 02 17:41:38.791522 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:38.791495 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n"] Nov 02 17:41:38.797664 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:38.797637 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:41:38.797812 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:38.797791 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:41:38.904132 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:38.904091 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/264e4e66-95ca-4b21-ad34-8e4317335082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b664d769-mw68n\" (UID: \"264e4e66-95ca-4b21-ad34-8e4317335082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n" Nov 02 17:41:38.904324 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:38.904146 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/264e4e66-95ca-4b21-ad34-8e4317335082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b664d769-mw68n\" (UID: \"264e4e66-95ca-4b21-ad34-8e4317335082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n" Nov 02 17:41:38.904324 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:38.904255 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trv2n\" (UniqueName: \"kubernetes.io/projected/264e4e66-95ca-4b21-ad34-8e4317335082-kube-api-access-trv2n\") pod \"maas-api-revoke-sub-b664d769-mw68n\" (UID: \"264e4e66-95ca-4b21-ad34-8e4317335082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n" Nov 02 17:41:39.005732 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:39.005694 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trv2n\" (UniqueName: \"kubernetes.io/projected/264e4e66-95ca-4b21-ad34-8e4317335082-kube-api-access-trv2n\") pod \"maas-api-revoke-sub-b664d769-mw68n\" (UID: \"264e4e66-95ca-4b21-ad34-8e4317335082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n" Nov 02 17:41:39.005949 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:39.005894 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/264e4e66-95ca-4b21-ad34-8e4317335082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b664d769-mw68n\" (UID: \"264e4e66-95ca-4b21-ad34-8e4317335082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n" Nov 02 17:41:39.006031 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:39.005953 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/264e4e66-95ca-4b21-ad34-8e4317335082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b664d769-mw68n\" (UID: \"264e4e66-95ca-4b21-ad34-8e4317335082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n" Nov 02 17:41:39.006586 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:39.006565 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/264e4e66-95ca-4b21-ad34-8e4317335082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b664d769-mw68n\" (UID: \"264e4e66-95ca-4b21-ad34-8e4317335082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n" Nov 02 17:41:39.008321 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:39.008297 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/264e4e66-95ca-4b21-ad34-8e4317335082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b664d769-mw68n\" (UID: \"264e4e66-95ca-4b21-ad34-8e4317335082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n" Nov 02 17:41:39.029694 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:39.029633 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trv2n\" (UniqueName: \"kubernetes.io/projected/264e4e66-95ca-4b21-ad34-8e4317335082-kube-api-access-trv2n\") pod \"maas-api-revoke-sub-b664d769-mw68n\" (UID: \"264e4e66-95ca-4b21-ad34-8e4317335082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n" Nov 02 17:41:39.097070 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:39.097045 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n" Nov 02 17:41:39.400282 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:39.400256 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n"] Nov 02 17:41:39.400427 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:41:39.400365 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264e4e66_95ca_4b21_ad34_8e4317335082.slice/crio-033761a8ec4341099849b78a40a9084a4330e9f87d6ff8b4f95d68a7d85185bc WatchSource:0}: Error finding container 033761a8ec4341099849b78a40a9084a4330e9f87d6ff8b4f95d68a7d85185bc: Status 404 returned error can't find the container with id 033761a8ec4341099849b78a40a9084a4330e9f87d6ff8b4f95d68a7d85185bc Nov 02 17:41:39.809138 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:39.809103 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n" event={"ID":"264e4e66-95ca-4b21-ad34-8e4317335082","Type":"ContainerStarted","Data":"5bb112a88bbfa2679f04631615505d996974439ad4100649abfd3197d3fca812"} Nov 02 17:41:39.809713 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:39.809161 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n" event={"ID":"264e4e66-95ca-4b21-ad34-8e4317335082","Type":"ContainerStarted","Data":"033761a8ec4341099849b78a40a9084a4330e9f87d6ff8b4f95d68a7d85185bc"} Nov 02 17:41:39.852287 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:39.852233 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n" podStartSLOduration=1.85221984 podStartE2EDuration="1.85221984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:39.85077973 +0000 UTC m=+718.472139506" watchObservedRunningTime="2026-11-02 17:41:39.85221984 +0000 UTC m=+718.473579616" Nov 02 17:41:40.817825 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:40.817790 4860 generic.go:356] "Generic (PLEG): container finished" podID="264e4e66-95ca-4b21-ad34-8e4317335082" containerID="5bb112a88bbfa2679f04631615505d996974439ad4100649abfd3197d3fca812" exitCode=0 Nov 02 17:41:40.818293 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:40.817878 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n" event={"ID":"264e4e66-95ca-4b21-ad34-8e4317335082","Type":"ContainerDied","Data":"5bb112a88bbfa2679f04631615505d996974439ad4100649abfd3197d3fca812"} Nov 02 17:41:42.014910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:42.014844 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n" Nov 02 17:41:42.139707 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:42.139618 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/264e4e66-95ca-4b21-ad34-8e4317335082-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/264e4e66-95ca-4b21-ad34-8e4317335082-maas-api-openshift-ca\") pod \"264e4e66-95ca-4b21-ad34-8e4317335082\" (UID: \"264e4e66-95ca-4b21-ad34-8e4317335082\") " Nov 02 17:41:42.139908 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:42.139817 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/264e4e66-95ca-4b21-ad34-8e4317335082-kube-api-access-trv2n\" (UniqueName: \"kubernetes.io/projected/264e4e66-95ca-4b21-ad34-8e4317335082-kube-api-access-trv2n\") pod \"264e4e66-95ca-4b21-ad34-8e4317335082\" (UID: \"264e4e66-95ca-4b21-ad34-8e4317335082\") " Nov 02 17:41:42.139984 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:42.139905 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/264e4e66-95ca-4b21-ad34-8e4317335082-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/264e4e66-95ca-4b21-ad34-8e4317335082-maas-api-xks-ca\") pod \"264e4e66-95ca-4b21-ad34-8e4317335082\" (UID: \"264e4e66-95ca-4b21-ad34-8e4317335082\") " Nov 02 17:41:42.140073 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:42.140041 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264e4e66-95ca-4b21-ad34-8e4317335082-maas-api-openshift-ca" pod "264e4e66-95ca-4b21-ad34-8e4317335082" (UID: "264e4e66-95ca-4b21-ad34-8e4317335082"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:42.140295 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:42.140275 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/264e4e66-95ca-4b21-ad34-8e4317335082-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:42.141932 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:42.141911 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/264e4e66-95ca-4b21-ad34-8e4317335082-maas-api-xks-ca" pod "264e4e66-95ca-4b21-ad34-8e4317335082" (UID: "264e4e66-95ca-4b21-ad34-8e4317335082"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:42.142051 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:42.142032 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264e4e66-95ca-4b21-ad34-8e4317335082-kube-api-access-trv2n" pod "264e4e66-95ca-4b21-ad34-8e4317335082" (UID: "264e4e66-95ca-4b21-ad34-8e4317335082"). InnerVolumeSpecName "kube-api-access-trv2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:42.241603 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:42.241569 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trv2n\" (UniqueName: \"kubernetes.io/projected/264e4e66-95ca-4b21-ad34-8e4317335082-kube-api-access-trv2n\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:42.241603 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:42.241600 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/264e4e66-95ca-4b21-ad34-8e4317335082-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:41:42.833240 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:42.833211 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n" Nov 02 17:41:42.833419 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:42.833239 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n" event={"ID":"264e4e66-95ca-4b21-ad34-8e4317335082","Type":"ContainerDied","Data":"033761a8ec4341099849b78a40a9084a4330e9f87d6ff8b4f95d68a7d85185bc"} Nov 02 17:41:42.833419 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:42.833288 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="033761a8ec4341099849b78a40a9084a4330e9f87d6ff8b4f95d68a7d85185bc" Nov 02 17:41:43.547111 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:43.547079 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:43.560049 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:43.560022 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:43.851158 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:43.851132 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:41:46.331839 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.331798 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462"] Nov 02 17:41:46.332377 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.332362 4860 cpu_manager.go:414] "containerMap: removing container" podUID="264e4e66-95ca-4b21-ad34-8e4317335082" containerName="revoke-keys" Nov 02 17:41:46.332420 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.332378 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="264e4e66-95ca-4b21-ad34-8e4317335082" containerName="revoke-keys" Nov 02 17:41:46.332482 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.332472 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="264e4e66-95ca-4b21-ad34-8e4317335082" containerName="revoke-keys" Nov 02 17:41:46.336229 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.336202 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.344593 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.344491 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p"] Nov 02 17:41:46.348328 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.348309 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.348656 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.348634 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm"] Nov 02 17:41:46.352712 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.352693 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.354407 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.354389 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-069fa5-openshift-default-dockercfg-ckxtj\"" Nov 02 17:41:46.366247 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.365690 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462"] Nov 02 17:41:46.383590 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.383561 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.383751 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.383665 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.383892 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.383865 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.384031 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.383936 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxztg\" (UniqueName: \"kubernetes.io/projected/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-kube-api-access-fxztg\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.384031 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.383970 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.384031 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.384007 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.384213 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.384068 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.384213 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.384106 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.384213 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.384137 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.388652 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.388214 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p"] Nov 02 17:41:46.400017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.399986 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm"] Nov 02 17:41:46.455903 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.455876 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f0a3b8-openshift-default-dockercfg-nch4k\"" Nov 02 17:41:46.455903 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.455904 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-863b87-openshift-default-dockercfg-5mwm4\"" Nov 02 17:41:46.485368 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.485328 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.485553 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.485471 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.485553 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.485525 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.485678 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.485587 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.485678 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.485623 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.485803 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.485762 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.485874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.485811 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.485972 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.485939 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxztg\" (UniqueName: \"kubernetes.io/projected/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-kube-api-access-fxztg\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.485972 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.485974 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.486135 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.486003 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.486135 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.486028 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtmqt\" (UniqueName: \"kubernetes.io/projected/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-kube-api-access-gtmqt\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.486135 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.486060 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.486135 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.486100 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.486135 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.486130 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.486380 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.486158 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.486380 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.486221 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.486496 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.486378 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knghs\" (UniqueName: \"kubernetes.io/projected/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-kube-api-access-knghs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.486496 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.486419 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.486496 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.486462 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.486685 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.486498 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.486685 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.486532 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.486685 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.486577 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.486685 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.486604 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.486685 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.486641 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.486685 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.486652 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.487093 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.486989 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.487093 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.487030 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.487246 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.487203 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.487246 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.487236 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.487341 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.487238 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.487341 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.487273 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.487341 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.487274 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.487957 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.487933 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.488249 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.488232 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.509571 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.509540 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxztg\" (UniqueName: \"kubernetes.io/projected/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-kube-api-access-fxztg\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.509818 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.509799 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.588189 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.588115 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.588189 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.588156 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knghs\" (UniqueName: \"kubernetes.io/projected/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-kube-api-access-knghs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.588357 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.588212 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.588357 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.588247 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.588357 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.588302 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.588456 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.588373 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.588456 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.588397 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.588456 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.588421 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.588614 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.588463 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.588614 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.588483 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.588729 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.588609 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.588786 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.588728 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.588880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.588842 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.588958 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.588919 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.588958 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.588924 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.588958 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.588930 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.589086 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.588962 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.589086 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.589002 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.589086 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.589032 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.589086 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.589056 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtmqt\" (UniqueName: \"kubernetes.io/projected/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-kube-api-access-gtmqt\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.589227 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.589106 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.589227 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.589132 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.589227 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.589176 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.589227 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.589210 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.589384 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.589272 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.589384 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.589273 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.589494 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.589470 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.590099 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.590075 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.591368 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.591338 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.591493 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.591472 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.591609 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.591521 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.591665 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.591654 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.611712 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.611683 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knghs\" (UniqueName: \"kubernetes.io/projected/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-kube-api-access-knghs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.611994 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.611974 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.612057 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.612001 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.612192 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.612175 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtmqt\" (UniqueName: \"kubernetes.io/projected/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-kube-api-access-gtmqt\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.652021 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.651989 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:46.687771 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.687748 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:46.691487 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.691459 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:46.972704 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:41:46.972676 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5e98eb_b605_4c58_aec6_e10ac2a890d6.slice/crio-9245de7f505c106738554807ec295ca0816b7e1448ae741f52634e7fb3a4e183 WatchSource:0}: Error finding container 9245de7f505c106738554807ec295ca0816b7e1448ae741f52634e7fb3a4e183: Status 404 returned error can't find the container with id 9245de7f505c106738554807ec295ca0816b7e1448ae741f52634e7fb3a4e183 Nov 02 17:41:46.973491 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.973465 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462"] Nov 02 17:41:46.975253 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.975219 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:41:46.975321 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.975284 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:41:46.975364 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:46.975317 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:41:47.018892 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:41:47.018845 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ed150c_7f9b_4422_b675_301dcb0c6e3f.slice/crio-e3e0134169d0cb22a39fcb8d4ff62b04f57fcf09c1b9f75d8ab74aa857d2c4ab WatchSource:0}: Error finding container e3e0134169d0cb22a39fcb8d4ff62b04f57fcf09c1b9f75d8ab74aa857d2c4ab: Status 404 returned error can't find the container with id e3e0134169d0cb22a39fcb8d4ff62b04f57fcf09c1b9f75d8ab74aa857d2c4ab Nov 02 17:41:47.019679 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:47.019655 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p"] Nov 02 17:41:47.021542 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:47.021510 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:41:47.021629 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:47.021570 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:41:47.021629 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:47.021597 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:41:47.048491 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:41:47.048464 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8a063b_3299_4bb8_a4d8_1eb79e6adca3.slice/crio-f865b73142b77b34c7a50fdabc7b3daff722014e2bdeb95b49c5218732cf8d04 WatchSource:0}: Error finding container f865b73142b77b34c7a50fdabc7b3daff722014e2bdeb95b49c5218732cf8d04: Status 404 returned error can't find the container with id f865b73142b77b34c7a50fdabc7b3daff722014e2bdeb95b49c5218732cf8d04 Nov 02 17:41:47.050037 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:47.050017 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm"] Nov 02 17:41:47.051284 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:47.051256 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:41:47.051360 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:47.051317 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:41:47.051360 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:47.051347 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:41:47.922025 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:47.921979 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" event={"ID":"c3ed150c-7f9b-4422-b675-301dcb0c6e3f","Type":"ContainerStarted","Data":"feaa572512091083c7abc3a83f224471f3642af2ec63ca1775b70364353790db"} Nov 02 17:41:47.922767 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:47.922648 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" event={"ID":"c3ed150c-7f9b-4422-b675-301dcb0c6e3f","Type":"ContainerStarted","Data":"e3e0134169d0cb22a39fcb8d4ff62b04f57fcf09c1b9f75d8ab74aa857d2c4ab"} Nov 02 17:41:47.934308 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:47.934255 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" event={"ID":"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3","Type":"ContainerStarted","Data":"ff641b2cf9a1dd6b53442dd633731a641f214cafc78ca1cd3fa26e8ac1693ae0"} Nov 02 17:41:47.934629 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:47.934579 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" event={"ID":"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3","Type":"ContainerStarted","Data":"f865b73142b77b34c7a50fdabc7b3daff722014e2bdeb95b49c5218732cf8d04"} Nov 02 17:41:47.937658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:47.937595 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" event={"ID":"fa5e98eb-b605-4c58-aec6-e10ac2a890d6","Type":"ContainerStarted","Data":"9fcf80adab1412996324f13f30f98a389492b9306cbb741ffbdcfdc23de17b41"} Nov 02 17:41:47.937658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:47.937634 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" event={"ID":"fa5e98eb-b605-4c58-aec6-e10ac2a890d6","Type":"ContainerStarted","Data":"9245de7f505c106738554807ec295ca0816b7e1448ae741f52634e7fb3a4e183"} Nov 02 17:41:48.102872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:48.102273 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" podStartSLOduration=2.102249965 podStartE2EDuration="2.102249965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:48.100242881 +0000 UTC m=+726.721602658" watchObservedRunningTime="2026-11-02 17:41:48.102249965 +0000 UTC m=+726.723609742" Nov 02 17:41:48.116874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:48.108759 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" podStartSLOduration=2.108742137 podStartE2EDuration="2.108742137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:48.001235972 +0000 UTC m=+726.622595757" watchObservedRunningTime="2026-11-02 17:41:48.108742137 +0000 UTC m=+726.730101914" Nov 02 17:41:48.149791 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:48.149727 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" podStartSLOduration=2.149713651 podStartE2EDuration="2.149713651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:48.148813957 +0000 UTC m=+726.770173733" watchObservedRunningTime="2026-11-02 17:41:48.149713651 +0000 UTC m=+726.771073426" Nov 02 17:41:48.652272 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:48.652186 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:48.690874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:48.685093 4860 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 17:41:48.690874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:48.685152 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" podUID="fa5e98eb-b605-4c58-aec6-e10ac2a890d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 17:41:48.690874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:48.689166 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:48.690874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:48.690645 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 17:41:48.690874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:48.690694 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" podUID="c3ed150c-7f9b-4422-b675-301dcb0c6e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 17:41:48.704873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:48.704223 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:48.715872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:48.707281 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:41:48.715872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:48.707331 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" podUID="dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 17:41:49.147685 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.147623 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg"] Nov 02 17:41:49.198873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.158510 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg"] Nov 02 17:41:49.198873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.158617 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.234880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.228920 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7df526\"/\"distinct-w6-7df526-kserve-self-signed-certs\"" Nov 02 17:41:49.308874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.308280 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-tls-certs\") pod \"distinct-w6-7df526-kserve-544d955d49-9whlg\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.308874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.308339 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-tmp-dir\") pod \"distinct-w6-7df526-kserve-544d955d49-9whlg\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.308874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.308370 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-model-cache\") pod \"distinct-w6-7df526-kserve-544d955d49-9whlg\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.308874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.308530 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-dshm\") pod \"distinct-w6-7df526-kserve-544d955d49-9whlg\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.308874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.308571 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-home\") pod \"distinct-w6-7df526-kserve-544d955d49-9whlg\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.308874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.308615 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqws9\" (UniqueName: \"kubernetes.io/projected/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-kube-api-access-vqws9\") pod \"distinct-w6-7df526-kserve-544d955d49-9whlg\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.428921 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.410668 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-dshm\") pod \"distinct-w6-7df526-kserve-544d955d49-9whlg\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.428921 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.410723 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-home\") pod \"distinct-w6-7df526-kserve-544d955d49-9whlg\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.428921 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.410765 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqws9\" (UniqueName: \"kubernetes.io/projected/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-kube-api-access-vqws9\") pod \"distinct-w6-7df526-kserve-544d955d49-9whlg\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.428921 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.410833 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-tls-certs\") pod \"distinct-w6-7df526-kserve-544d955d49-9whlg\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.428921 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.410883 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-tmp-dir\") pod \"distinct-w6-7df526-kserve-544d955d49-9whlg\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.428921 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.410909 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-model-cache\") pod \"distinct-w6-7df526-kserve-544d955d49-9whlg\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.428921 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.411387 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-model-cache\") pod \"distinct-w6-7df526-kserve-544d955d49-9whlg\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.428921 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.422726 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-home\") pod \"distinct-w6-7df526-kserve-544d955d49-9whlg\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.472877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.430397 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-tmp-dir\") pod \"distinct-w6-7df526-kserve-544d955d49-9whlg\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.472877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.461256 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-tls-certs\") pod \"distinct-w6-7df526-kserve-544d955d49-9whlg\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.520590 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.520490 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-dshm\") pod \"distinct-w6-7df526-kserve-544d955d49-9whlg\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.537921 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.537641 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqws9\" (UniqueName: \"kubernetes.io/projected/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-kube-api-access-vqws9\") pod \"distinct-w6-7df526-kserve-544d955d49-9whlg\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.654355 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.653095 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:49.654355 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.654317 4860 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 17:41:49.654604 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.654473 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" podUID="fa5e98eb-b605-4c58-aec6-e10ac2a890d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 17:41:49.699922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.697443 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:41:49.699922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.697503 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" podUID="dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 17:41:49.699922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.697817 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 17:41:49.699922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:49.697879 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" podUID="c3ed150c-7f9b-4422-b675-301dcb0c6e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 17:41:50.369075 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:50.367059 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg"] Nov 02 17:41:50.657938 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:50.653352 4860 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 17:41:50.657938 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:50.653422 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" podUID="fa5e98eb-b605-4c58-aec6-e10ac2a890d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 17:41:50.693013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:50.692897 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 17:41:50.693013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:50.692950 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" podUID="c3ed150c-7f9b-4422-b675-301dcb0c6e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 17:41:50.693880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:50.693344 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:41:50.693880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:50.693386 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" podUID="dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 17:41:51.093875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:51.091513 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" event={"ID":"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5","Type":"ContainerStarted","Data":"a65c79f3e530878efb797ed99d60140f0c380257673787b01bb59cc13afb0228"} Nov 02 17:41:51.093875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:51.091573 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" event={"ID":"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5","Type":"ContainerStarted","Data":"07724339e84c540c8a2a8f8b07af3700096040897b6a9e301b3c24d0b90f9905"} Nov 02 17:41:51.141459 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:51.141393 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podStartSLOduration=2.1413733600000002 podStartE2EDuration="2.14137336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:51.13836673 +0000 UTC m=+729.759726500" watchObservedRunningTime="2026-11-02 17:41:51.14137336 +0000 UTC m=+729.762733137" Nov 02 17:41:51.652685 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:51.652631 4860 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 17:41:51.653102 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:51.652697 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" podUID="fa5e98eb-b605-4c58-aec6-e10ac2a890d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 17:41:51.662878 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:51.658943 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:51.688567 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:51.688530 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 17:41:51.688841 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:51.688812 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" podUID="c3ed150c-7f9b-4422-b675-301dcb0c6e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 17:41:51.692224 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:51.692199 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:41:51.692942 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:51.692913 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" podUID="dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 17:41:51.842205 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:41:51.842141 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:41:52.652433 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:52.652397 4860 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 17:41:52.652612 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:52.652457 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" podUID="fa5e98eb-b605-4c58-aec6-e10ac2a890d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 17:41:52.691352 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:52.691316 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 17:41:52.691933 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:52.691900 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" podUID="c3ed150c-7f9b-4422-b675-301dcb0c6e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 17:41:52.692733 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:52.692709 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:41:52.692930 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:52.692908 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" podUID="dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 17:41:53.653745 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:53.653133 4860 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 17:41:53.653745 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:53.653200 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" podUID="fa5e98eb-b605-4c58-aec6-e10ac2a890d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 17:41:53.688200 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:53.688158 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 17:41:53.688418 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:53.688223 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" podUID="c3ed150c-7f9b-4422-b675-301dcb0c6e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 17:41:53.692560 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:53.692527 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:41:53.693016 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:53.692587 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" podUID="dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 17:41:54.657752 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.657711 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:54.658001 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.657976 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:54.658653 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.658628 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 17:41:54.692303 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.692275 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:54.692566 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.692544 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:54.693197 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.693174 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:41:54.695944 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.695928 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:54.696186 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.696169 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:54.696728 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.696711 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 17:41:54.750578 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.750534 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65"] Nov 02 17:41:54.755439 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.755410 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:54.774762 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.774727 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"distinct-w0-069fa5-kserve-self-signed-certs\"" Nov 02 17:41:54.778136 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.778112 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65"] Nov 02 17:41:54.876521 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.876473 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b70b32-bbca-4658-97e0-e8c0e7927307-tls-certs\") pod \"distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:54.878551 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.878516 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-tmp-dir\") pod \"distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:54.878861 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.878820 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl9qf\" (UniqueName: \"kubernetes.io/projected/76b70b32-bbca-4658-97e0-e8c0e7927307-kube-api-access-rl9qf\") pod \"distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:54.878975 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.878949 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-dshm\") pod \"distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:54.879044 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.878981 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-model-cache\") pod \"distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:54.879044 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.879023 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-home\") pod \"distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:54.980839 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.980760 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl9qf\" (UniqueName: \"kubernetes.io/projected/76b70b32-bbca-4658-97e0-e8c0e7927307-kube-api-access-rl9qf\") pod \"distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:54.980839 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.980811 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-dshm\") pod \"distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:54.980839 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.980829 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-model-cache\") pod \"distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:54.980839 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.980868 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-home\") pod \"distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:54.981179 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.980971 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b70b32-bbca-4658-97e0-e8c0e7927307-tls-certs\") pod \"distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:54.981179 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.981062 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-tmp-dir\") pod \"distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:54.981872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.981374 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-model-cache\") pod \"distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:54.981872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.981696 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-home\") pod \"distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:54.982181 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.982142 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-tmp-dir\") pod \"distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:54.983370 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.983326 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-dshm\") pod \"distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:54.983932 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:54.983912 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b70b32-bbca-4658-97e0-e8c0e7927307-tls-certs\") pod \"distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:55.022881 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:55.022613 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl9qf\" (UniqueName: \"kubernetes.io/projected/76b70b32-bbca-4658-97e0-e8c0e7927307-kube-api-access-rl9qf\") pod \"distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:55.070596 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:55.070559 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:55.392175 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:41:55.392139 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76b70b32_bbca_4658_97e0_e8c0e7927307.slice/crio-6aabc715ffb6732b81670b2d57fc72a617013c665033c55e74e757c7bf595a2f WatchSource:0}: Error finding container 6aabc715ffb6732b81670b2d57fc72a617013c665033c55e74e757c7bf595a2f: Status 404 returned error can't find the container with id 6aabc715ffb6732b81670b2d57fc72a617013c665033c55e74e757c7bf595a2f Nov 02 17:41:55.393390 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:55.393362 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65"] Nov 02 17:41:56.186363 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:56.186322 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" event={"ID":"76b70b32-bbca-4658-97e0-e8c0e7927307","Type":"ContainerStarted","Data":"b7662f9d1ef269d306e89d58541dbe51a4c94714d5799bad4bbb00233092ac6e"} Nov 02 17:41:56.186363 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:56.186367 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" event={"ID":"76b70b32-bbca-4658-97e0-e8c0e7927307","Type":"ContainerStarted","Data":"6aabc715ffb6732b81670b2d57fc72a617013c665033c55e74e757c7bf595a2f"} Nov 02 17:41:56.231769 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:56.231711 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podStartSLOduration=2.23169729 podStartE2EDuration="2.23169729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:56.228464183 +0000 UTC m=+734.849823959" watchObservedRunningTime="2026-11-02 17:41:56.23169729 +0000 UTC m=+734.853057065" Nov 02 17:41:57.070945 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:57.070911 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:41:59.654839 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:59.654808 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:41:59.668024 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:41:59.667996 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:42:00.236884 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:00.236839 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:42:01.160215 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.160178 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh"] Nov 02 17:42:01.164424 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.164405 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.181990 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.181969 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5eedab-openshift-default-dockercfg-57c6m\"" Nov 02 17:42:01.198620 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.198590 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh"] Nov 02 17:42:01.278344 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.278315 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.278514 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.278364 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnqdr\" (UniqueName: \"kubernetes.io/projected/c92d9c87-7ba0-4d6e-b659-9e916730c79e-kube-api-access-xnqdr\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.278514 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.278469 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.278592 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.278519 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.278628 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.278613 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.278660 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.278637 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.278690 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.278660 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.278769 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.278755 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.278837 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.278826 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.379680 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.379642 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.379880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.379687 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.379880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.379717 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnqdr\" (UniqueName: \"kubernetes.io/projected/c92d9c87-7ba0-4d6e-b659-9e916730c79e-kube-api-access-xnqdr\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.379880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.379756 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.379880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.379795 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.379880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.379844 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.380181 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.379900 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.380181 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.379933 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.380181 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.379988 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.380345 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.380181 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.380345 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.380269 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.380454 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.380384 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.380454 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.380404 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.380534 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.380508 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.382362 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.382329 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.382545 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.382528 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.403395 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.403368 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.403589 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.403525 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnqdr\" (UniqueName: \"kubernetes.io/projected/c92d9c87-7ba0-4d6e-b659-9e916730c79e-kube-api-access-xnqdr\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.479305 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.479207 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:01.812384 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:42:01.812353 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92d9c87_7ba0_4d6e_b659_9e916730c79e.slice/crio-ec5f0a01cb754f983839856f989c58efba71b821216c7ab113c7c472df504185 WatchSource:0}: Error finding container ec5f0a01cb754f983839856f989c58efba71b821216c7ab113c7c472df504185: Status 404 returned error can't find the container with id ec5f0a01cb754f983839856f989c58efba71b821216c7ab113c7c472df504185 Nov 02 17:42:01.812965 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.812937 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh"] Nov 02 17:42:01.815367 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.815329 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:42:01.815465 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.815412 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:42:01.815465 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:01.815448 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:42:02.245523 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.245130 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" event={"ID":"c92d9c87-7ba0-4d6e-b659-9e916730c79e","Type":"ContainerStarted","Data":"71d866d15c5ebd732f0d33a1bda18a78921026df493328a1a5e9f5fab9f1a050"} Nov 02 17:42:02.245523 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.245232 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" event={"ID":"c92d9c87-7ba0-4d6e-b659-9e916730c79e","Type":"ContainerStarted","Data":"ec5f0a01cb754f983839856f989c58efba71b821216c7ab113c7c472df504185"} Nov 02 17:42:02.296080 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.295966 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" podStartSLOduration=1.29595159 podStartE2EDuration="1.29595159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:02.290588778 +0000 UTC m=+740.911948555" watchObservedRunningTime="2026-11-02 17:42:02.29595159 +0000 UTC m=+740.917311366" Nov 02 17:42:02.482121 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.482087 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:02.482510 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.482487 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 17:42:02.482659 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.482630 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" podUID="c92d9c87-7ba0-4d6e-b659-9e916730c79e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 17:42:02.698025 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.694672 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79"] Nov 02 17:42:02.700320 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.700290 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79"] Nov 02 17:42:02.700475 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.700402 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:02.720077 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.720043 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"kube-root-ca.crt\"" Nov 02 17:42:02.720221 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.720194 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"distinct-w5-863b87-kserve-self-signed-certs\"" Nov 02 17:42:02.720339 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.720321 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"default-dockercfg-h47km\"" Nov 02 17:42:02.721423 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.721399 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"openshift-service-ca.crt\"" Nov 02 17:42:02.796661 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.796627 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-dshm\") pod \"distinct-w5-863b87-kserve-77c7998f8b-x8p79\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:02.796826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.796666 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-model-cache\") pod \"distinct-w5-863b87-kserve-77c7998f8b-x8p79\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:02.796826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.796697 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-home\") pod \"distinct-w5-863b87-kserve-77c7998f8b-x8p79\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:02.796965 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.796881 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-tmp-dir\") pod \"distinct-w5-863b87-kserve-77c7998f8b-x8p79\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:02.797110 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.797090 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/355a0269-1933-46ee-b701-5a5c45da4061-tls-certs\") pod \"distinct-w5-863b87-kserve-77c7998f8b-x8p79\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:02.797173 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.797144 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq9f5\" (UniqueName: \"kubernetes.io/projected/355a0269-1933-46ee-b701-5a5c45da4061-kube-api-access-wq9f5\") pod \"distinct-w5-863b87-kserve-77c7998f8b-x8p79\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:02.901872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.900424 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-tmp-dir\") pod \"distinct-w5-863b87-kserve-77c7998f8b-x8p79\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:02.901872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.901035 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-tmp-dir\") pod \"distinct-w5-863b87-kserve-77c7998f8b-x8p79\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:02.905925 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.901635 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/355a0269-1933-46ee-b701-5a5c45da4061-tls-certs\") pod \"distinct-w5-863b87-kserve-77c7998f8b-x8p79\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:02.905925 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.902659 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq9f5\" (UniqueName: \"kubernetes.io/projected/355a0269-1933-46ee-b701-5a5c45da4061-kube-api-access-wq9f5\") pod \"distinct-w5-863b87-kserve-77c7998f8b-x8p79\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:02.905925 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.902750 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-dshm\") pod \"distinct-w5-863b87-kserve-77c7998f8b-x8p79\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:02.905925 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.902779 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-model-cache\") pod \"distinct-w5-863b87-kserve-77c7998f8b-x8p79\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:02.905925 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.902816 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-home\") pod \"distinct-w5-863b87-kserve-77c7998f8b-x8p79\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:02.905925 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.903147 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-model-cache\") pod \"distinct-w5-863b87-kserve-77c7998f8b-x8p79\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:02.905925 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.904201 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-home\") pod \"distinct-w5-863b87-kserve-77c7998f8b-x8p79\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:02.917961 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.917265 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-dshm\") pod \"distinct-w5-863b87-kserve-77c7998f8b-x8p79\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:02.921873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.918417 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/355a0269-1933-46ee-b701-5a5c45da4061-tls-certs\") pod \"distinct-w5-863b87-kserve-77c7998f8b-x8p79\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:02.938696 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:02.938657 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq9f5\" (UniqueName: \"kubernetes.io/projected/355a0269-1933-46ee-b701-5a5c45da4061-kube-api-access-wq9f5\") pod \"distinct-w5-863b87-kserve-77c7998f8b-x8p79\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:03.014873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:03.014755 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:03.383444 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:03.383400 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79"] Nov 02 17:42:03.385429 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:42:03.385397 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod355a0269_1933_46ee_b701_5a5c45da4061.slice/crio-f331f78513996eb0d17ac6da039162ef5665db066c8a069fa553963140749d09 WatchSource:0}: Error finding container f331f78513996eb0d17ac6da039162ef5665db066c8a069fa553963140749d09: Status 404 returned error can't find the container with id f331f78513996eb0d17ac6da039162ef5665db066c8a069fa553963140749d09 Nov 02 17:42:03.484875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:03.481272 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 17:42:03.484875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:03.481329 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" podUID="c92d9c87-7ba0-4d6e-b659-9e916730c79e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 17:42:04.287686 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:04.287639 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" event={"ID":"355a0269-1933-46ee-b701-5a5c45da4061","Type":"ContainerStarted","Data":"d43bc000ddb3cf0e3e21882f2466225022fe58942b8ce5c2e7168baf4e36fbc3"} Nov 02 17:42:04.287839 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:04.287696 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" event={"ID":"355a0269-1933-46ee-b701-5a5c45da4061","Type":"ContainerStarted","Data":"f331f78513996eb0d17ac6da039162ef5665db066c8a069fa553963140749d09"} Nov 02 17:42:04.351473 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:04.351417 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podStartSLOduration=2.351403032 podStartE2EDuration="2.351403032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:04.347748311 +0000 UTC m=+742.969108086" watchObservedRunningTime="2026-11-02 17:42:04.351403032 +0000 UTC m=+742.972762808" Nov 02 17:42:04.479885 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:04.479731 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 17:42:04.479885 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:04.479837 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" podUID="c92d9c87-7ba0-4d6e-b659-9e916730c79e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 17:42:05.015536 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:05.015478 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:05.077875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:05.077693 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:42:05.091195 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:05.091172 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:42:05.306319 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:05.306297 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:42:05.480275 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:05.480238 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 17:42:05.480909 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:05.480873 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" podUID="c92d9c87-7ba0-4d6e-b659-9e916730c79e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 17:42:06.480220 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:06.480185 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 17:42:06.480431 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:06.480274 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" podUID="c92d9c87-7ba0-4d6e-b659-9e916730c79e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 17:42:07.482582 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:07.482541 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 17:42:07.483083 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:07.482619 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" podUID="c92d9c87-7ba0-4d6e-b659-9e916730c79e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 17:42:08.606431 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:08.606401 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:08.607347 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:08.607320 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:08.608369 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:08.608345 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 17:42:13.015613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:13.015578 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:13.028078 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:13.028053 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:13.377191 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:13.377159 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:42:16.328200 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.328146 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq"] Nov 02 17:42:16.341913 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.339380 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.355931 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.351731 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq"] Nov 02 17:42:16.355931 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.355717 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2899ab-openshift-default-dockercfg-t6vss\"" Nov 02 17:42:16.389881 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.387653 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57a3d284-8cce-4c08-830a-605040cfb458-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.389881 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.388020 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.389881 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.388096 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a3d284-8cce-4c08-830a-605040cfb458-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.389881 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.388330 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.389881 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.389125 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv7tx\" (UniqueName: \"kubernetes.io/projected/57a3d284-8cce-4c08-830a-605040cfb458-kube-api-access-kv7tx\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.389881 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.389191 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.389881 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.389242 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.389881 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.389271 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.389881 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.389346 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a3d284-8cce-4c08-830a-605040cfb458-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.491154 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.491108 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.491332 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.491168 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.491332 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.491231 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a3d284-8cce-4c08-830a-605040cfb458-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.491332 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.491290 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57a3d284-8cce-4c08-830a-605040cfb458-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.491439 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.491331 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.491439 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.491368 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a3d284-8cce-4c08-830a-605040cfb458-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.491528 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.491510 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.491567 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.491554 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv7tx\" (UniqueName: \"kubernetes.io/projected/57a3d284-8cce-4c08-830a-605040cfb458-kube-api-access-kv7tx\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.491629 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.491614 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.492167 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.492135 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.492579 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.492554 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a3d284-8cce-4c08-830a-605040cfb458-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.492971 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.492949 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.493332 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.493311 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.493665 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.493641 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.495393 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.495346 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.497227 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.497204 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a3d284-8cce-4c08-830a-605040cfb458-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.518511 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.518479 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57a3d284-8cce-4c08-830a-605040cfb458-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.528015 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.527983 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv7tx\" (UniqueName: \"kubernetes.io/projected/57a3d284-8cce-4c08-830a-605040cfb458-kube-api-access-kv7tx\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:16.687363 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:16.687267 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:17.139875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:17.139606 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq"] Nov 02 17:42:17.155664 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:17.153413 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:42:17.155664 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:17.153522 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:42:17.155664 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:17.153586 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:42:17.438875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:17.438087 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" event={"ID":"57a3d284-8cce-4c08-830a-605040cfb458","Type":"ContainerStarted","Data":"c768723eed2d9671919cabffd8d7c67420b6798c9fa51a06f7ae49ab1488fe71"} Nov 02 17:42:17.527183 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:17.524546 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" podStartSLOduration=1.524510495 podStartE2EDuration="1.524510495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:17.507727235 +0000 UTC m=+756.129087012" watchObservedRunningTime="2026-11-02 17:42:17.524510495 +0000 UTC m=+756.145870271" Nov 02 17:42:17.688919 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:17.688801 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:18.448549 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:18.448507 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" event={"ID":"57a3d284-8cce-4c08-830a-605040cfb458","Type":"ContainerStarted","Data":"189d517dfe3103337930a18e894cf7c35cba210420c6375c0c54fc085592bd96"} Nov 02 17:42:18.689692 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:18.689652 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 17:42:18.689891 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:18.689719 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" podUID="57a3d284-8cce-4c08-830a-605040cfb458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 17:42:19.688574 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:19.688535 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 17:42:19.688979 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:19.688601 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" podUID="57a3d284-8cce-4c08-830a-605040cfb458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 17:42:20.688347 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:20.688313 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 17:42:20.688534 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:20.688366 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" podUID="57a3d284-8cce-4c08-830a-605040cfb458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 17:42:21.688408 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:21.688371 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 17:42:21.688805 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:21.688434 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" podUID="57a3d284-8cce-4c08-830a-605040cfb458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 17:42:21.790427 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:42:21.790384 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:42:22.688170 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:22.688132 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 17:42:22.688350 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:22.688188 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" podUID="57a3d284-8cce-4c08-830a-605040cfb458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 17:42:23.689070 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:23.689032 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 17:42:23.689474 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:23.689094 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" podUID="57a3d284-8cce-4c08-830a-605040cfb458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 17:42:24.692954 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:24.692924 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:24.693418 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:24.693399 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:24.694184 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:24.694162 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:42:28.143710 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.143675 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52"] Nov 02 17:42:28.148585 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.148560 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.168418 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.168387 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52"] Nov 02 17:42:28.174407 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.174381 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2899ab\"/\"distinct-2-w4-2899ab-kserve-self-signed-certs\"" Nov 02 17:42:28.203629 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.203603 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf880079-551f-4973-ab8b-4009619a4fe5-tls-certs\") pod \"distinct-2-w4-2899ab-kserve-77c8984777-plj52\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.203778 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.203657 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-tmp-dir\") pod \"distinct-2-w4-2899ab-kserve-77c8984777-plj52\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.203778 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.203757 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98kms\" (UniqueName: \"kubernetes.io/projected/cf880079-551f-4973-ab8b-4009619a4fe5-kube-api-access-98kms\") pod \"distinct-2-w4-2899ab-kserve-77c8984777-plj52\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.203898 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.203784 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-dshm\") pod \"distinct-2-w4-2899ab-kserve-77c8984777-plj52\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.203898 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.203834 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-model-cache\") pod \"distinct-2-w4-2899ab-kserve-77c8984777-plj52\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.204017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.203976 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-home\") pod \"distinct-2-w4-2899ab-kserve-77c8984777-plj52\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.305440 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.305406 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-model-cache\") pod \"distinct-2-w4-2899ab-kserve-77c8984777-plj52\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.305637 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.305471 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-home\") pod \"distinct-2-w4-2899ab-kserve-77c8984777-plj52\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.305704 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.305651 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf880079-551f-4973-ab8b-4009619a4fe5-tls-certs\") pod \"distinct-2-w4-2899ab-kserve-77c8984777-plj52\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.305704 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.305694 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-tmp-dir\") pod \"distinct-2-w4-2899ab-kserve-77c8984777-plj52\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.305829 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.305722 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98kms\" (UniqueName: \"kubernetes.io/projected/cf880079-551f-4973-ab8b-4009619a4fe5-kube-api-access-98kms\") pod \"distinct-2-w4-2899ab-kserve-77c8984777-plj52\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.305829 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.305745 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-dshm\") pod \"distinct-2-w4-2899ab-kserve-77c8984777-plj52\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.306717 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.306686 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-tmp-dir\") pod \"distinct-2-w4-2899ab-kserve-77c8984777-plj52\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.307333 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.307302 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-model-cache\") pod \"distinct-2-w4-2899ab-kserve-77c8984777-plj52\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.308252 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.308225 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-home\") pod \"distinct-2-w4-2899ab-kserve-77c8984777-plj52\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.308691 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.308665 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-dshm\") pod \"distinct-2-w4-2899ab-kserve-77c8984777-plj52\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.309198 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.309178 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf880079-551f-4973-ab8b-4009619a4fe5-tls-certs\") pod \"distinct-2-w4-2899ab-kserve-77c8984777-plj52\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.352123 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.352091 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98kms\" (UniqueName: \"kubernetes.io/projected/cf880079-551f-4973-ab8b-4009619a4fe5-kube-api-access-98kms\") pod \"distinct-2-w4-2899ab-kserve-77c8984777-plj52\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.464798 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.464716 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:28.779924 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:42:28.779892 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf880079_551f_4973_ab8b_4009619a4fe5.slice/crio-8cff816b4ff9fdb77dd2c8989b7e9217173c77fc93ab7ed9c27d08832c87dfd0 WatchSource:0}: Error finding container 8cff816b4ff9fdb77dd2c8989b7e9217173c77fc93ab7ed9c27d08832c87dfd0: Status 404 returned error can't find the container with id 8cff816b4ff9fdb77dd2c8989b7e9217173c77fc93ab7ed9c27d08832c87dfd0 Nov 02 17:42:28.780677 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:28.780654 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52"] Nov 02 17:42:29.588499 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:29.588463 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" event={"ID":"cf880079-551f-4973-ab8b-4009619a4fe5","Type":"ContainerStarted","Data":"01401ae9f88c12f436dd91b4f3b25280bc4247ef59e053aa5c69d0dc4a1f7877"} Nov 02 17:42:29.588499 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:29.588513 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" event={"ID":"cf880079-551f-4973-ab8b-4009619a4fe5","Type":"ContainerStarted","Data":"8cff816b4ff9fdb77dd2c8989b7e9217173c77fc93ab7ed9c27d08832c87dfd0"} Nov 02 17:42:29.633663 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:29.633614 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podStartSLOduration=1.633600404 podStartE2EDuration="1.633600404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:29.630165459 +0000 UTC m=+768.251525232" watchObservedRunningTime="2026-11-02 17:42:29.633600404 +0000 UTC m=+768.254960180" Nov 02 17:42:30.464860 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:30.464827 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:38.465047 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:38.465009 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:38.478095 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:38.478065 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:38.682279 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:38.682239 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:42:46.344027 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.343985 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp"] Nov 02 17:42:46.351253 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.351222 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.354052 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.354019 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95"] Nov 02 17:42:46.358488 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.358462 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.361374 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.361345 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz"] Nov 02 17:42:46.366216 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.366192 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.387033 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.386973 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95"] Nov 02 17:42:46.397902 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.397863 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp"] Nov 02 17:42:46.407915 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.407883 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz"] Nov 02 17:42:46.452808 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.452776 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.453019 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.452812 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.453019 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.452885 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.453019 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.452946 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.453019 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.452985 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.453200 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453064 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.453200 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453131 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.453200 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453151 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.453343 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453216 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.453343 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453260 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.453343 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453330 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.453485 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453400 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.453485 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453429 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5pd4\" (UniqueName: \"kubernetes.io/projected/0102a8d6-f853-4e95-aed3-3e033d85e970-kube-api-access-x5pd4\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.453485 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453471 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0102a8d6-f853-4e95-aed3-3e033d85e970-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.453592 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453548 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.453630 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453600 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.453663 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453649 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.453696 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453670 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj4d2\" (UniqueName: \"kubernetes.io/projected/5a72d837-5288-4398-8bd5-3f4b9db3002f-kube-api-access-fj4d2\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.453696 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453688 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.453778 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453727 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.453812 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453787 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a72d837-5288-4398-8bd5-3f4b9db3002f-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.453865 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453829 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz5vd\" (UniqueName: \"kubernetes.io/projected/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-kube-api-access-rz5vd\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.453901 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453883 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.453960 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453931 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.453960 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.453956 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.454037 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.454008 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.454080 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.454054 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.555738 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.555702 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0102a8d6-f853-4e95-aed3-3e033d85e970-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.555986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.555746 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.555986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.555769 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.555986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.555803 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.555986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.555825 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj4d2\" (UniqueName: \"kubernetes.io/projected/5a72d837-5288-4398-8bd5-3f4b9db3002f-kube-api-access-fj4d2\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.555986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.555843 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.555986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.555891 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.555986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.555959 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a72d837-5288-4398-8bd5-3f4b9db3002f-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.556352 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556016 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz5vd\" (UniqueName: \"kubernetes.io/projected/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-kube-api-access-rz5vd\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.556352 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556048 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.556352 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556119 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.556352 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556149 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.556352 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556197 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.556352 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556245 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.556352 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556261 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.556352 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556278 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.556352 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556290 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.556869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556404 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.556869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556553 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.556869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556607 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.556869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556648 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.556869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556693 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.556869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556768 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.556869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556826 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0102a8d6-f853-4e95-aed3-3e033d85e970-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.556869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556866 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.557291 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556944 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.557291 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556954 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a72d837-5288-4398-8bd5-3f4b9db3002f-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.557291 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.556987 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.557291 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.557041 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.557291 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.557074 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.557291 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.557090 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.557291 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.557101 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.557291 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.557110 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.557291 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.557148 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.557291 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.557158 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.557291 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.557177 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.557291 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.557209 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.557291 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.557254 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5pd4\" (UniqueName: \"kubernetes.io/projected/0102a8d6-f853-4e95-aed3-3e033d85e970-kube-api-access-x5pd4\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.557991 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.557355 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.557991 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.557387 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.557991 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.557614 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.557991 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.557913 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.559575 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.559501 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.559840 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.559794 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.560047 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.560023 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.560237 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.560216 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.560513 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.560492 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.560754 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.560738 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.579768 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.579732 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.579983 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.579898 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.580537 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.580509 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz5vd\" (UniqueName: \"kubernetes.io/projected/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-kube-api-access-rz5vd\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.581410 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.581383 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj4d2\" (UniqueName: \"kubernetes.io/projected/5a72d837-5288-4398-8bd5-3f4b9db3002f-kube-api-access-fj4d2\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.583813 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.583787 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.583918 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.583885 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5pd4\" (UniqueName: \"kubernetes.io/projected/0102a8d6-f853-4e95-aed3-3e033d85e970-kube-api-access-x5pd4\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:46.671997 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.671903 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:46.699472 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.699444 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:46.704875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:46.704832 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:47.027885 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.027263 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp"] Nov 02 17:42:47.030012 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:42:47.029979 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a72d837_5288_4398_8bd5_3f4b9db3002f.slice/crio-a2ae21d43b08b519d5cfee0bc3105c28dbbd9367a6603d56bf904e5f9c10cf9a WatchSource:0}: Error finding container a2ae21d43b08b519d5cfee0bc3105c28dbbd9367a6603d56bf904e5f9c10cf9a: Status 404 returned error can't find the container with id a2ae21d43b08b519d5cfee0bc3105c28dbbd9367a6603d56bf904e5f9c10cf9a Nov 02 17:42:47.033035 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.032999 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:42:47.033131 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.033069 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:42:47.033131 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.033100 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:42:47.057269 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.057238 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95"] Nov 02 17:42:47.057441 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:42:47.057417 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3078bbb_3ccf_46f7_9a76_bc9ff80995f2.slice/crio-1fb6f0722c35f3411570a76087a4f100e85df162fd9393c748e0d2b6ab5c1eb7 WatchSource:0}: Error finding container 1fb6f0722c35f3411570a76087a4f100e85df162fd9393c748e0d2b6ab5c1eb7: Status 404 returned error can't find the container with id 1fb6f0722c35f3411570a76087a4f100e85df162fd9393c748e0d2b6ab5c1eb7 Nov 02 17:42:47.060167 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.060135 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:42:47.060256 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.060200 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:42:47.060256 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.060228 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:42:47.069955 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.069927 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz"] Nov 02 17:42:47.073669 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:42:47.073626 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0102a8d6_f853_4e95_aed3_3e033d85e970.slice/crio-7ae6fdfae31a4fe0f12ae4822b218f391fe611c0831aa9be8fa16488cc4585fc WatchSource:0}: Error finding container 7ae6fdfae31a4fe0f12ae4822b218f391fe611c0831aa9be8fa16488cc4585fc: Status 404 returned error can't find the container with id 7ae6fdfae31a4fe0f12ae4822b218f391fe611c0831aa9be8fa16488cc4585fc Nov 02 17:42:47.077039 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.077006 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:42:47.077140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.077071 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:42:47.077140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.077103 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:42:47.747378 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.747339 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" event={"ID":"5a72d837-5288-4398-8bd5-3f4b9db3002f","Type":"ContainerStarted","Data":"952e39c2d290df01b3c1afbb93d9ea5256dd327962c75ead5d81bb19426d0197"} Nov 02 17:42:47.747885 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.747402 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" event={"ID":"5a72d837-5288-4398-8bd5-3f4b9db3002f","Type":"ContainerStarted","Data":"a2ae21d43b08b519d5cfee0bc3105c28dbbd9367a6603d56bf904e5f9c10cf9a"} Nov 02 17:42:47.749178 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.749145 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" event={"ID":"0102a8d6-f853-4e95-aed3-3e033d85e970","Type":"ContainerStarted","Data":"d5f65253ddf3706b952f384952c36e618fba85411689c0b99a87558698ace0ed"} Nov 02 17:42:47.749320 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.749198 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" event={"ID":"0102a8d6-f853-4e95-aed3-3e033d85e970","Type":"ContainerStarted","Data":"7ae6fdfae31a4fe0f12ae4822b218f391fe611c0831aa9be8fa16488cc4585fc"} Nov 02 17:42:47.750720 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.750697 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" event={"ID":"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2","Type":"ContainerStarted","Data":"5666cf0e22c229c8e8f59e13b44f11436e7c48f43a2cb4e490afff0172dd7364"} Nov 02 17:42:47.750877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.750743 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" event={"ID":"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2","Type":"ContainerStarted","Data":"1fb6f0722c35f3411570a76087a4f100e85df162fd9393c748e0d2b6ab5c1eb7"} Nov 02 17:42:47.795289 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.795232 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" podStartSLOduration=1.795216962 podStartE2EDuration="1.795216962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:47.793537348 +0000 UTC m=+786.414897124" watchObservedRunningTime="2026-11-02 17:42:47.795216962 +0000 UTC m=+786.416576738" Nov 02 17:42:47.846002 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.845933 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" podStartSLOduration=1.845907756 podStartE2EDuration="1.845907756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:47.839401381 +0000 UTC m=+786.460761156" watchObservedRunningTime="2026-11-02 17:42:47.845907756 +0000 UTC m=+786.467267535" Nov 02 17:42:47.902336 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:47.902265 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" podStartSLOduration=1.902247929 podStartE2EDuration="1.902247929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:47.900572601 +0000 UTC m=+786.521932380" watchObservedRunningTime="2026-11-02 17:42:47.902247929 +0000 UTC m=+786.523607705" Nov 02 17:42:48.690489 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:48.684145 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:48.693917 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:48.690900 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:42:48.693917 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:48.690987 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" podUID="5a72d837-5288-4398-8bd5-3f4b9db3002f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 17:42:48.715905 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:48.706235 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:48.715905 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:48.706559 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:48.716882 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:48.716757 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:42:48.720886 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:48.717049 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" podUID="b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 17:42:48.720886 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:48.717182 4860 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 17:42:48.720886 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:48.717217 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" podUID="0102a8d6-f853-4e95-aed3-3e033d85e970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 17:42:49.152018 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.151148 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc"] Nov 02 17:42:49.274870 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.234131 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:49.274870 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.273674 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"distinct-2-w0-069fa5-kserve-self-signed-certs\"" Nov 02 17:42:49.286872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.283237 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxdjb\" (UniqueName: \"kubernetes.io/projected/fe8fb265-b2fa-44a3-8960-834a02db7553-kube-api-access-zxdjb\") pod \"distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:49.286872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.283339 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8fb265-b2fa-44a3-8960-834a02db7553-tls-certs\") pod \"distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:49.286872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.283410 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-tmp-dir\") pod \"distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:49.286872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.283437 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-home\") pod \"distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:49.286872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.283538 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-dshm\") pod \"distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:49.286872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.283635 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-model-cache\") pod \"distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:49.397910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.387790 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-tmp-dir\") pod \"distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:49.397910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.387836 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-home\") pod \"distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:49.397910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.387971 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-dshm\") pod \"distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:49.397910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.388084 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-model-cache\") pod \"distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:49.397910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.388150 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxdjb\" (UniqueName: \"kubernetes.io/projected/fe8fb265-b2fa-44a3-8960-834a02db7553-kube-api-access-zxdjb\") pod \"distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:49.397910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.390400 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-home\") pod \"distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:49.435922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.402038 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-tmp-dir\") pod \"distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:49.435922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.402380 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-model-cache\") pod \"distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:49.435922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.429287 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8fb265-b2fa-44a3-8960-834a02db7553-tls-certs\") pod \"distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:49.437658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.437588 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc"] Nov 02 17:42:49.490660 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.490623 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxdjb\" (UniqueName: \"kubernetes.io/projected/fe8fb265-b2fa-44a3-8960-834a02db7553-kube-api-access-zxdjb\") pod \"distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:49.597868 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.593303 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8fb265-b2fa-44a3-8960-834a02db7553-tls-certs\") pod \"distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:49.604620 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.604547 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-dshm\") pod \"distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:49.672897 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.672843 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:42:49.673187 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.673162 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" podUID="5a72d837-5288-4398-8bd5-3f4b9db3002f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 17:42:49.700248 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.700161 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:42:49.700516 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.700490 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" podUID="b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 17:42:49.705183 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.705155 4860 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 17:42:49.705395 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.705367 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" podUID="0102a8d6-f853-4e95-aed3-3e033d85e970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 17:42:49.744592 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:49.744560 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:50.245894 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:50.241726 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 17:42:50.649320 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:50.649288 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc"] Nov 02 17:42:50.679920 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:50.679277 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:42:50.679920 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:50.679354 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" podUID="5a72d837-5288-4398-8bd5-3f4b9db3002f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 17:42:50.700757 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:50.700717 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:42:50.704963 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:50.700780 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" podUID="b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 17:42:50.711908 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:50.706164 4860 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 17:42:50.711908 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:50.706222 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" podUID="0102a8d6-f853-4e95-aed3-3e033d85e970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 17:42:50.947881 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:50.947822 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" event={"ID":"fe8fb265-b2fa-44a3-8960-834a02db7553","Type":"ContainerStarted","Data":"40166d0742079a1e4235fd84ba14bf8b9e8384137cbe691833f4a576728d974f"} Nov 02 17:42:51.705872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:51.705223 4860 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 17:42:51.705872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:51.705287 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" podUID="0102a8d6-f853-4e95-aed3-3e033d85e970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 17:42:51.706458 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:51.705822 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:42:51.706458 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:51.705970 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" podUID="b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 17:42:51.751124 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:51.751063 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:42:51.751335 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:51.751161 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" podUID="5a72d837-5288-4398-8bd5-3f4b9db3002f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 17:42:52.042960 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:52.042601 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" event={"ID":"fe8fb265-b2fa-44a3-8960-834a02db7553","Type":"ContainerStarted","Data":"d94f88a65fd5635737b33220a7c3d145621277bb49f96f0b0d3e50d548dbe73c"} Nov 02 17:42:52.049459 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:42:52.049420 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:42:52.174876 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:52.173701 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podStartSLOduration=3.173680194 podStartE2EDuration="3.173680194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:52.171293001 +0000 UTC m=+790.792652778" watchObservedRunningTime="2026-11-02 17:42:52.173680194 +0000 UTC m=+790.795039971" Nov 02 17:42:52.689875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:52.680343 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:42:52.689875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:52.680409 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" podUID="5a72d837-5288-4398-8bd5-3f4b9db3002f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 17:42:52.729914 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:52.704773 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:42:52.729914 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:52.704831 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" podUID="b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 17:42:52.733875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:52.731710 4860 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 17:42:52.733875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:52.731759 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" podUID="0102a8d6-f853-4e95-aed3-3e033d85e970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 17:42:52.746020 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:52.745947 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:53.709871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:53.680897 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:42:53.720740 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:53.715938 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" podUID="5a72d837-5288-4398-8bd5-3f4b9db3002f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 17:42:53.732977 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:53.732378 4860 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 17:42:53.732977 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:53.732435 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" podUID="0102a8d6-f853-4e95-aed3-3e033d85e970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 17:42:53.732977 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:53.732784 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:42:53.732977 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:53.732827 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" podUID="b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 17:42:54.672875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:54.672739 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:42:54.672875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:54.672800 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" podUID="5a72d837-5288-4398-8bd5-3f4b9db3002f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 17:42:55.157871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:55.157697 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:55.169895 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:55.160088 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:55.169895 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:55.161183 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:42:55.391963 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:55.391928 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:55.393841 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:55.393816 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:55.402357 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:55.402300 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:42:55.852240 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:55.852117 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:55.853372 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:55.852892 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:55.853963 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:55.853939 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:42:59.746024 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:59.745993 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:42:59.759969 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:42:59.759943 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:43:00.168134 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:00.168100 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:43:05.526482 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.526446 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6"] Nov 02 17:43:05.532065 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.532043 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:05.551583 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.551552 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6"] Nov 02 17:43:05.557076 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.557052 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"kube-root-ca.crt\"" Nov 02 17:43:05.557244 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.557075 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"openshift-service-ca.crt\"" Nov 02 17:43:05.557344 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.557082 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"default-dockercfg-q6667\"" Nov 02 17:43:05.559684 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.559664 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"distinct-2-w3-5eedab-kserve-self-signed-certs\"" Nov 02 17:43:05.691800 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.691764 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-tls-certs\") pod \"distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:05.692005 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.691986 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mkch\" (UniqueName: \"kubernetes.io/projected/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-kube-api-access-8mkch\") pod \"distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:05.692075 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.692036 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-model-cache\") pod \"distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:05.692184 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.692155 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-dshm\") pod \"distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:05.692238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.692223 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-home\") pod \"distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:05.692306 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.692290 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-tmp-dir\") pod \"distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:05.813703 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.803363 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-tls-certs\") pod \"distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:05.819288 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.819254 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mkch\" (UniqueName: \"kubernetes.io/projected/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-kube-api-access-8mkch\") pod \"distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:05.819520 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.819503 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-model-cache\") pod \"distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:05.819647 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.819632 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-dshm\") pod \"distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:05.819763 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.819750 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-home\") pod \"distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:05.819905 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.819892 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-tmp-dir\") pod \"distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:05.820604 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.820528 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-tmp-dir\") pod \"distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:05.821373 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.821328 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-model-cache\") pod \"distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:05.832637 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.823994 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-home\") pod \"distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:05.833929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.832804 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-dshm\") pod \"distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:05.838113 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.838072 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-tls-certs\") pod \"distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:05.846617 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.846589 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mkch\" (UniqueName: \"kubernetes.io/projected/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-kube-api-access-8mkch\") pod \"distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:05.847354 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:05.847328 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:06.180186 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:43:06.180144 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7df1cfb_533d_4c2b_ad3c_1dbec34b0388.slice/crio-f8574df84943aeda43f97474bb29ea12877aed954e9954a706659dd24371e202 WatchSource:0}: Error finding container f8574df84943aeda43f97474bb29ea12877aed954e9954a706659dd24371e202: Status 404 returned error can't find the container with id f8574df84943aeda43f97474bb29ea12877aed954e9954a706659dd24371e202 Nov 02 17:43:06.180360 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:06.180338 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6"] Nov 02 17:43:06.215311 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:06.215272 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" event={"ID":"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388","Type":"ContainerStarted","Data":"f8574df84943aeda43f97474bb29ea12877aed954e9954a706659dd24371e202"} Nov 02 17:43:07.229507 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:07.229461 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" event={"ID":"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388","Type":"ContainerStarted","Data":"c2894fa0d6b5c8c09177b91e0b1f67cfda42a12ce15f90904d7361df59b9c80a"} Nov 02 17:43:07.279238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:07.279191 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podStartSLOduration=2.279178666 podStartE2EDuration="2.279178666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:07.273539232 +0000 UTC m=+805.894899008" watchObservedRunningTime="2026-11-02 17:43:07.279178666 +0000 UTC m=+805.900538442" Nov 02 17:43:07.848133 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:07.848097 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:15.848288 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:15.848256 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:15.861054 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:15.861025 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:16.322425 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.322388 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:16.484980 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.484946 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm"] Nov 02 17:43:16.489601 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.489576 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.514492 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.514455 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm"] Nov 02 17:43:16.581921 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.581818 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.582061 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.581927 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.582061 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.581950 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.582061 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.581996 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.582168 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.582131 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wpls\" (UniqueName: \"kubernetes.io/projected/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-kube-api-access-7wpls\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.582203 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.582171 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.582243 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.582231 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.582319 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.582302 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.582397 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.582386 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.684153 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.684117 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wpls\" (UniqueName: \"kubernetes.io/projected/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-kube-api-access-7wpls\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.684153 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.684158 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.684352 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.684330 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.684441 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.684423 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.684541 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.684527 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.684591 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.684577 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.684636 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.684621 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.684689 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.684653 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.684689 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.684677 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.684903 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.684880 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.684903 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.684896 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.685068 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.684977 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.685133 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.685106 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.685267 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.685243 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.686705 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.686683 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.686911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.686896 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.708531 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.708504 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.708699 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.708680 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wpls\" (UniqueName: \"kubernetes.io/projected/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-kube-api-access-7wpls\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:16.810186 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:16.810148 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:17.136198 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:43:17.136166 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6eb0e0c_ec0c_4cf6_bf46_972c912fd7c4.slice/crio-67e2237cf2b1cce58f1b4f80120b677040928ddb1a94582876b9f0bed69858d3 WatchSource:0}: Error finding container 67e2237cf2b1cce58f1b4f80120b677040928ddb1a94582876b9f0bed69858d3: Status 404 returned error can't find the container with id 67e2237cf2b1cce58f1b4f80120b677040928ddb1a94582876b9f0bed69858d3 Nov 02 17:43:17.137509 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:17.137485 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm"] Nov 02 17:43:17.139029 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:17.138989 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:43:17.139143 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:17.139075 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:43:17.139143 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:17.139121 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:43:17.322597 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:17.322557 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" event={"ID":"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4","Type":"ContainerStarted","Data":"b6e188e7917a333690af41ed1f45f82985e8f79e89d4104ea56412bd9bb1937a"} Nov 02 17:43:17.322788 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:17.322626 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" event={"ID":"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4","Type":"ContainerStarted","Data":"67e2237cf2b1cce58f1b4f80120b677040928ddb1a94582876b9f0bed69858d3"} Nov 02 17:43:17.374574 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:17.374514 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" podStartSLOduration=1.374501132 podStartE2EDuration="1.374501132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:17.370384821 +0000 UTC m=+815.991744598" watchObservedRunningTime="2026-11-02 17:43:17.374501132 +0000 UTC m=+815.995860907" Nov 02 17:43:17.810978 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:17.810929 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:18.812527 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:18.812492 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 17:43:18.812974 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:18.812544 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" podUID="a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 17:43:19.002582 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:19.002545 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z"] Nov 02 17:43:19.007436 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:19.007410 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z" Nov 02 17:43:19.028807 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:19.028777 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z"] Nov 02 17:43:19.036131 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:19.036105 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:43:19.036266 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:19.036221 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:43:19.115353 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:19.115266 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pffw\" (UniqueName: \"kubernetes.io/projected/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-kube-api-access-5pffw\") pod \"maas-api-revoke-sub-27c0bb11-m5q9z\" (UID: \"eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z" Nov 02 17:43:19.115501 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:19.115362 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27c0bb11-m5q9z\" (UID: \"eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z" Nov 02 17:43:19.115561 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:19.115546 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27c0bb11-m5q9z\" (UID: \"eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z" Nov 02 17:43:19.217120 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:19.217085 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27c0bb11-m5q9z\" (UID: \"eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z" Nov 02 17:43:19.217317 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:19.217291 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pffw\" (UniqueName: \"kubernetes.io/projected/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-kube-api-access-5pffw\") pod \"maas-api-revoke-sub-27c0bb11-m5q9z\" (UID: \"eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z" Nov 02 17:43:19.217388 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:19.217366 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27c0bb11-m5q9z\" (UID: \"eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z" Nov 02 17:43:19.217833 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:19.217808 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27c0bb11-m5q9z\" (UID: \"eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z" Nov 02 17:43:19.220485 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:19.220454 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27c0bb11-m5q9z\" (UID: \"eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z" Nov 02 17:43:19.241509 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:19.241480 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pffw\" (UniqueName: \"kubernetes.io/projected/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-kube-api-access-5pffw\") pod \"maas-api-revoke-sub-27c0bb11-m5q9z\" (UID: \"eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z" Nov 02 17:43:19.322671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:19.322637 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z" Nov 02 17:43:19.652479 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:43:19.652449 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeacd9905_aa4b_4e8d_ba8c_bd16c6791bfd.slice/crio-015d0185324cc9eb002334a511cb469d3cb5bc82f9d68d6c8eb1afc24f524025 WatchSource:0}: Error finding container 015d0185324cc9eb002334a511cb469d3cb5bc82f9d68d6c8eb1afc24f524025: Status 404 returned error can't find the container with id 015d0185324cc9eb002334a511cb469d3cb5bc82f9d68d6c8eb1afc24f524025 Nov 02 17:43:19.677443 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:19.654723 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z"] Nov 02 17:43:19.810934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:19.810900 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 17:43:19.811111 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:19.810949 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" podUID="a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 17:43:20.351827 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:20.351789 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z" event={"ID":"eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd","Type":"ContainerStarted","Data":"14fcd77ca9fa079a506abe2f3c7d6294b73a17709096784f978fc03710415cc1"} Nov 02 17:43:20.352237 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:20.351888 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z" event={"ID":"eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd","Type":"ContainerStarted","Data":"015d0185324cc9eb002334a511cb469d3cb5bc82f9d68d6c8eb1afc24f524025"} Nov 02 17:43:20.408321 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:20.408265 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z" podStartSLOduration=2.40825233 podStartE2EDuration="2.40825233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:20.404694507 +0000 UTC m=+819.026054283" watchObservedRunningTime="2026-11-02 17:43:20.40825233 +0000 UTC m=+819.029612142" Nov 02 17:43:20.810910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:20.810883 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 17:43:20.811040 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:20.810940 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" podUID="a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 17:43:21.387269 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:21.387235 4860 generic.go:356] "Generic (PLEG): container finished" podID="eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd" containerID="14fcd77ca9fa079a506abe2f3c7d6294b73a17709096784f978fc03710415cc1" exitCode=0 Nov 02 17:43:21.387730 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:21.387315 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z" event={"ID":"eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd","Type":"ContainerDied","Data":"14fcd77ca9fa079a506abe2f3c7d6294b73a17709096784f978fc03710415cc1"} Nov 02 17:43:21.788686 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:43:21.788592 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:43:21.811137 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:21.811105 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 17:43:21.811316 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:21.811161 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" podUID="a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 17:43:22.628121 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:22.628093 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z" Nov 02 17:43:22.805933 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:22.805897 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-maas-api-openshift-ca\") pod \"eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd\" (UID: \"eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd\") " Nov 02 17:43:22.806096 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:22.806060 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-kube-api-access-5pffw\" (UniqueName: \"kubernetes.io/projected/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-kube-api-access-5pffw\") pod \"eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd\" (UID: \"eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd\") " Nov 02 17:43:22.806246 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:22.806229 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-maas-api-xks-ca\") pod \"eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd\" (UID: \"eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd\") " Nov 02 17:43:22.806301 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:22.806271 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-maas-api-openshift-ca" pod "eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd" (UID: "eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:43:22.807874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:22.807539 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:43:22.818103 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:22.818056 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 17:43:22.818274 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:22.818123 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" podUID="a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 17:43:22.821576 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:22.821543 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-maas-api-xks-ca" pod "eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd" (UID: "eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:43:22.823229 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:22.823183 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-kube-api-access-5pffw" pod "eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd" (UID: "eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd"). InnerVolumeSpecName "kube-api-access-5pffw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:22.912259 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:22.912222 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:43:22.912430 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:22.912262 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pffw\" (UniqueName: \"kubernetes.io/projected/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd-kube-api-access-5pffw\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:43:23.409603 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:23.409566 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z" event={"ID":"eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd","Type":"ContainerDied","Data":"015d0185324cc9eb002334a511cb469d3cb5bc82f9d68d6c8eb1afc24f524025"} Nov 02 17:43:23.409996 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:23.409972 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="015d0185324cc9eb002334a511cb469d3cb5bc82f9d68d6c8eb1afc24f524025" Nov 02 17:43:23.410134 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:23.409658 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z" Nov 02 17:43:23.810721 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:23.810688 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 17:43:23.811156 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:23.810747 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" podUID="a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 17:43:24.815111 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:24.815081 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:24.815575 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:24.815554 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:24.816285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:24.816264 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 17:43:27.013161 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:27.013129 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb"] Nov 02 17:43:27.013873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:27.013842 4860 cpu_manager.go:414] "containerMap: removing container" podUID="eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd" containerName="revoke-keys" Nov 02 17:43:27.013923 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:27.013874 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd" containerName="revoke-keys" Nov 02 17:43:27.013991 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:27.013982 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd" containerName="revoke-keys" Nov 02 17:43:27.017646 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:27.017629 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb" Nov 02 17:43:27.043753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:27.043724 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb"] Nov 02 17:43:27.051726 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:27.051701 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:43:27.051839 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:27.051809 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:43:27.082304 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:27.082272 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e97ccd-b641-413d-8aab-8f56f5c0e51d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01ec2929-w7skb\" (UID: \"21e97ccd-b641-413d-8aab-8f56f5c0e51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb" Nov 02 17:43:27.082438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:27.082317 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e97ccd-b641-413d-8aab-8f56f5c0e51d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01ec2929-w7skb\" (UID: \"21e97ccd-b641-413d-8aab-8f56f5c0e51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb" Nov 02 17:43:27.082438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:27.082349 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfzf4\" (UniqueName: \"kubernetes.io/projected/21e97ccd-b641-413d-8aab-8f56f5c0e51d-kube-api-access-dfzf4\") pod \"maas-api-revoke-sub-01ec2929-w7skb\" (UID: \"21e97ccd-b641-413d-8aab-8f56f5c0e51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb" Nov 02 17:43:27.184074 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:27.184036 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e97ccd-b641-413d-8aab-8f56f5c0e51d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01ec2929-w7skb\" (UID: \"21e97ccd-b641-413d-8aab-8f56f5c0e51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb" Nov 02 17:43:27.184074 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:27.184075 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e97ccd-b641-413d-8aab-8f56f5c0e51d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01ec2929-w7skb\" (UID: \"21e97ccd-b641-413d-8aab-8f56f5c0e51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb" Nov 02 17:43:27.184291 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:27.184092 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfzf4\" (UniqueName: \"kubernetes.io/projected/21e97ccd-b641-413d-8aab-8f56f5c0e51d-kube-api-access-dfzf4\") pod \"maas-api-revoke-sub-01ec2929-w7skb\" (UID: \"21e97ccd-b641-413d-8aab-8f56f5c0e51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb" Nov 02 17:43:27.184870 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:27.184833 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e97ccd-b641-413d-8aab-8f56f5c0e51d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01ec2929-w7skb\" (UID: \"21e97ccd-b641-413d-8aab-8f56f5c0e51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb" Nov 02 17:43:27.186571 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:27.186543 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e97ccd-b641-413d-8aab-8f56f5c0e51d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01ec2929-w7skb\" (UID: \"21e97ccd-b641-413d-8aab-8f56f5c0e51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb" Nov 02 17:43:27.209125 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:27.209087 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfzf4\" (UniqueName: \"kubernetes.io/projected/21e97ccd-b641-413d-8aab-8f56f5c0e51d-kube-api-access-dfzf4\") pod \"maas-api-revoke-sub-01ec2929-w7skb\" (UID: \"21e97ccd-b641-413d-8aab-8f56f5c0e51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb" Nov 02 17:43:27.332071 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:27.332042 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb" Nov 02 17:43:27.645969 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:27.645937 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb"] Nov 02 17:43:27.646090 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:43:27.645983 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e97ccd_b641_413d_8aab_8f56f5c0e51d.slice/crio-c9eacfd562abaec953943b78bd32ea47beaa00a31bfa9fb6493fd8920f68ffd9 WatchSource:0}: Error finding container c9eacfd562abaec953943b78bd32ea47beaa00a31bfa9fb6493fd8920f68ffd9: Status 404 returned error can't find the container with id c9eacfd562abaec953943b78bd32ea47beaa00a31bfa9fb6493fd8920f68ffd9 Nov 02 17:43:28.463076 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:28.463036 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb" event={"ID":"21e97ccd-b641-413d-8aab-8f56f5c0e51d","Type":"ContainerStarted","Data":"27c6064c268abe285052971d82a9a9950cce9ef3d0227d6ebb857069771832f5"} Nov 02 17:43:28.463573 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:28.463118 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb" event={"ID":"21e97ccd-b641-413d-8aab-8f56f5c0e51d","Type":"ContainerStarted","Data":"c9eacfd562abaec953943b78bd32ea47beaa00a31bfa9fb6493fd8920f68ffd9"} Nov 02 17:43:28.512111 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:28.512051 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb" podStartSLOduration=2.512036563 podStartE2EDuration="2.512036563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:28.505741304 +0000 UTC m=+827.127101081" watchObservedRunningTime="2026-11-02 17:43:28.512036563 +0000 UTC m=+827.133396362" Nov 02 17:43:29.472988 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:29.472957 4860 generic.go:356] "Generic (PLEG): container finished" podID="21e97ccd-b641-413d-8aab-8f56f5c0e51d" containerID="27c6064c268abe285052971d82a9a9950cce9ef3d0227d6ebb857069771832f5" exitCode=0 Nov 02 17:43:29.473372 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:29.473044 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb" event={"ID":"21e97ccd-b641-413d-8aab-8f56f5c0e51d","Type":"ContainerDied","Data":"27c6064c268abe285052971d82a9a9950cce9ef3d0227d6ebb857069771832f5"} Nov 02 17:43:30.642518 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:30.642488 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb" Nov 02 17:43:30.732328 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:30.732191 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e97ccd-b641-413d-8aab-8f56f5c0e51d-kube-api-access-dfzf4\" (UniqueName: \"kubernetes.io/projected/21e97ccd-b641-413d-8aab-8f56f5c0e51d-kube-api-access-dfzf4\") pod \"21e97ccd-b641-413d-8aab-8f56f5c0e51d\" (UID: \"21e97ccd-b641-413d-8aab-8f56f5c0e51d\") " Nov 02 17:43:30.732328 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:30.732234 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21e97ccd-b641-413d-8aab-8f56f5c0e51d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e97ccd-b641-413d-8aab-8f56f5c0e51d-maas-api-xks-ca\") pod \"21e97ccd-b641-413d-8aab-8f56f5c0e51d\" (UID: \"21e97ccd-b641-413d-8aab-8f56f5c0e51d\") " Nov 02 17:43:30.732328 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:30.732286 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21e97ccd-b641-413d-8aab-8f56f5c0e51d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e97ccd-b641-413d-8aab-8f56f5c0e51d-maas-api-openshift-ca\") pod \"21e97ccd-b641-413d-8aab-8f56f5c0e51d\" (UID: \"21e97ccd-b641-413d-8aab-8f56f5c0e51d\") " Nov 02 17:43:30.733050 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:30.733000 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e97ccd-b641-413d-8aab-8f56f5c0e51d-maas-api-openshift-ca" pod "21e97ccd-b641-413d-8aab-8f56f5c0e51d" (UID: "21e97ccd-b641-413d-8aab-8f56f5c0e51d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:43:30.734738 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:30.734699 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e97ccd-b641-413d-8aab-8f56f5c0e51d-kube-api-access-dfzf4" pod "21e97ccd-b641-413d-8aab-8f56f5c0e51d" (UID: "21e97ccd-b641-413d-8aab-8f56f5c0e51d"). InnerVolumeSpecName "kube-api-access-dfzf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:30.736650 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:30.736618 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21e97ccd-b641-413d-8aab-8f56f5c0e51d-maas-api-xks-ca" pod "21e97ccd-b641-413d-8aab-8f56f5c0e51d" (UID: "21e97ccd-b641-413d-8aab-8f56f5c0e51d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:43:30.833779 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:30.833737 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfzf4\" (UniqueName: \"kubernetes.io/projected/21e97ccd-b641-413d-8aab-8f56f5c0e51d-kube-api-access-dfzf4\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:43:30.833779 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:30.833770 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e97ccd-b641-413d-8aab-8f56f5c0e51d-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:43:30.833779 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:30.833779 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e97ccd-b641-413d-8aab-8f56f5c0e51d-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:43:31.143376 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.143341 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8"] Nov 02 17:43:31.144025 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.144010 4860 cpu_manager.go:414] "containerMap: removing container" podUID="21e97ccd-b641-413d-8aab-8f56f5c0e51d" containerName="revoke-keys" Nov 02 17:43:31.144067 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.144027 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e97ccd-b641-413d-8aab-8f56f5c0e51d" containerName="revoke-keys" Nov 02 17:43:31.144141 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.144132 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e97ccd-b641-413d-8aab-8f56f5c0e51d" containerName="revoke-keys" Nov 02 17:43:31.147893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.147877 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.226568 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.226535 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8"] Nov 02 17:43:31.238023 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.237988 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebc4571-d5c2-45f7-80de-5f79648303f4-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.238185 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.238086 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p542m\" (UniqueName: \"kubernetes.io/projected/0ebc4571-d5c2-45f7-80de-5f79648303f4-kube-api-access-p542m\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.238185 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.238155 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.238185 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.238174 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.238376 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.238335 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.238431 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.238393 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.238489 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.238476 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.238545 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.238505 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.238593 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.238545 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.340231 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.340190 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebc4571-d5c2-45f7-80de-5f79648303f4-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.340390 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.340239 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p542m\" (UniqueName: \"kubernetes.io/projected/0ebc4571-d5c2-45f7-80de-5f79648303f4-kube-api-access-p542m\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.340390 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.340302 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.340390 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.340328 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.340390 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.340386 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.340636 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.340438 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.340636 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.340547 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.340745 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.340662 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.340745 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.340732 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.340880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.340750 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.340880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.340792 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.340976 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.340906 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.340976 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.340963 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.341098 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.341074 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebc4571-d5c2-45f7-80de-5f79648303f4-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.342993 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.342964 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.343184 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.343168 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.389526 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.389498 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p542m\" (UniqueName: \"kubernetes.io/projected/0ebc4571-d5c2-45f7-80de-5f79648303f4-kube-api-access-p542m\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.402089 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.402033 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.463387 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.463355 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:31.497043 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.497000 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb" Nov 02 17:43:31.502230 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.502201 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb" event={"ID":"21e97ccd-b641-413d-8aab-8f56f5c0e51d","Type":"ContainerDied","Data":"c9eacfd562abaec953943b78bd32ea47beaa00a31bfa9fb6493fd8920f68ffd9"} Nov 02 17:43:31.502364 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.502253 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9eacfd562abaec953943b78bd32ea47beaa00a31bfa9fb6493fd8920f68ffd9" Nov 02 17:43:31.795067 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:43:31.795037 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ebc4571_d5c2_45f7_80de_5f79648303f4.slice/crio-c5cf8b42508774ab4cd6f57e004b8f6b2d18cf50b75e8c6b73d7c2fc29810ca4 WatchSource:0}: Error finding container c5cf8b42508774ab4cd6f57e004b8f6b2d18cf50b75e8c6b73d7c2fc29810ca4: Status 404 returned error can't find the container with id c5cf8b42508774ab4cd6f57e004b8f6b2d18cf50b75e8c6b73d7c2fc29810ca4 Nov 02 17:43:31.795608 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.795585 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8"] Nov 02 17:43:31.798206 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.798167 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:43:31.798322 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.798250 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:43:31.798322 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:31.798287 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:43:32.505150 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:32.505112 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" event={"ID":"0ebc4571-d5c2-45f7-80de-5f79648303f4","Type":"ContainerStarted","Data":"6b6c82d53d03ce3c28d7c2f39df0f2fdb88747cce90319002e49e4b86c77f9b5"} Nov 02 17:43:32.505354 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:32.505175 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" event={"ID":"0ebc4571-d5c2-45f7-80de-5f79648303f4","Type":"ContainerStarted","Data":"c5cf8b42508774ab4cd6f57e004b8f6b2d18cf50b75e8c6b73d7c2fc29810ca4"} Nov 02 17:43:32.586872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:32.586300 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" podStartSLOduration=1.586278055 podStartE2EDuration="1.586278055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:32.571490212 +0000 UTC m=+831.192849995" watchObservedRunningTime="2026-11-02 17:43:32.586278055 +0000 UTC m=+831.207637831" Nov 02 17:43:33.476874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:33.474039 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:33.476874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:33.474265 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:43:33.476874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:33.474314 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" podUID="0ebc4571-d5c2-45f7-80de-5f79648303f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 17:43:34.474874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:34.474249 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:43:34.474874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:34.474329 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" podUID="0ebc4571-d5c2-45f7-80de-5f79648303f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 17:43:35.471256 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:35.471218 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:43:35.471938 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:35.471906 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" podUID="0ebc4571-d5c2-45f7-80de-5f79648303f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 17:43:36.475870 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:36.468453 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:43:36.475870 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:36.468533 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" podUID="0ebc4571-d5c2-45f7-80de-5f79648303f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 17:43:37.342879 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:37.338933 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:37.466842 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:37.466807 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:43:37.467728 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:37.467123 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" podUID="0ebc4571-d5c2-45f7-80de-5f79648303f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 17:43:37.563010 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:37.562958 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:37.645071 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:37.644957 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:37.676982 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:37.676931 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:37.681973 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:37.681936 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 02 17:43:37.730883 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:37.730818 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:37.852999 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:37.851944 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:37.866956 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:37.866917 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 02 17:43:37.948050 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:37.947951 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.161870 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.160990 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.285951 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.239058 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.388821 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.386994 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 02 17:43:38.415259 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.406034 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 02 17:43:38.415259 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.407752 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.415259 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.408073 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:38.415259 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.409449 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.490885 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.490610 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:43:38.490885 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.490699 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" podUID="0ebc4571-d5c2-45f7-80de-5f79648303f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 17:43:38.630949 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.630909 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.631946 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.631923 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:43:38.655737 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.655684 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.667134 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.656118 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:43:38.681349 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.681307 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.681911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.681882 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:43:38.682985 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.682558 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 02 17:43:38.682985 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.682880 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:43:38.853142 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.852965 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.901937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.875146 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:43:38.901937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.875218 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:43:38.901937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.883315 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:43:38.901937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.883788 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.901937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.884127 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:43:38.929485 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.916205 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:38.929485 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.916597 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:43:38.949168 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.949056 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 02 17:43:38.949810 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:38.949755 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:43:39.276974 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.259256 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:39.276974 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.260536 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:43:39.284963 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.282809 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:39.284963 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.283521 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:43:39.353029 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.325754 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:43:39.353029 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.336926 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:43:39.440875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.417897 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:39.508875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.478407 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:39.508875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.478692 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:43:39.508875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.479759 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:39.508875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.480002 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:43:39.508875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.488206 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:43:39.508875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.488250 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" podUID="0ebc4571-d5c2-45f7-80de-5f79648303f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 17:43:39.689037 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.688996 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:39.691221 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.691186 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:39.707438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.707299 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:43:39.707438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.707350 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" podUID="c92d9c87-7ba0-4d6e-b659-9e916730c79e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:39.925251 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.921136 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:39.959170 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.959078 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:40.048481 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:39.982952 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:43:40.048750 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:40.048720 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" podUID="b53fb7b2-d91c-409b-bf83-e24b95a7608f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:40.048878 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:40.032014 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:43:40.108875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:40.090082 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:40.392987 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:40.336407 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:40.392987 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:40.339993 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:43:40.392987 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:40.342774 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:40.392987 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:40.354625 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:40.392987 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:40.390965 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:40.621875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:40.615419 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 02 17:43:40.673952 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:40.646736 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:43:40.734932 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:40.674428 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:40.736797 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:40.699993 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:43:40.737235 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:40.737196 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" podUID="0ebc4571-d5c2-45f7-80de-5f79648303f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 17:43:40.796629 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:40.796575 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 02 17:43:40.828017 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:40.827970 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:40.864918 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:40.864882 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:43:40.865195 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:40.865149 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" podUID="57a3d284-8cce-4c08-830a-605040cfb458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:40.866064 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:40.866033 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:43:40.866257 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:40.866223 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" podUID="dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:41.040960 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.028381 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:43:41.046620 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.046594 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:43:41.048717 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.048680 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:41.064323 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.063501 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:43:41.089957 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.089926 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:43:41.090816 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.090795 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:43:41.110086 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.110061 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:43:41.127956 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.127925 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm"] Nov 02 17:43:41.135614 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.135586 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm"] Nov 02 17:43:41.135790 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.135692 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm" Nov 02 17:43:41.155030 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.154996 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:43:41.156962 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.156941 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:43:41.268708 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.268668 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbc8f\" (UniqueName: \"kubernetes.io/projected/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-kube-api-access-hbc8f\") pod \"maas-api-revoke-sub-8cbeb6cc-2nzxm\" (UID: \"2783bfae-7a20-40d8-bc9b-52a3de2f97f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm" Nov 02 17:43:41.269052 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.269022 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cbeb6cc-2nzxm\" (UID: \"2783bfae-7a20-40d8-bc9b-52a3de2f97f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm" Nov 02 17:43:41.269190 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.269133 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cbeb6cc-2nzxm\" (UID: \"2783bfae-7a20-40d8-bc9b-52a3de2f97f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm" Nov 02 17:43:41.371173 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.371138 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cbeb6cc-2nzxm\" (UID: \"2783bfae-7a20-40d8-bc9b-52a3de2f97f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm" Nov 02 17:43:41.371352 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.371302 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbc8f\" (UniqueName: \"kubernetes.io/projected/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-kube-api-access-hbc8f\") pod \"maas-api-revoke-sub-8cbeb6cc-2nzxm\" (UID: \"2783bfae-7a20-40d8-bc9b-52a3de2f97f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm" Nov 02 17:43:41.371489 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.371473 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cbeb6cc-2nzxm\" (UID: \"2783bfae-7a20-40d8-bc9b-52a3de2f97f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm" Nov 02 17:43:41.371969 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.371942 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cbeb6cc-2nzxm\" (UID: \"2783bfae-7a20-40d8-bc9b-52a3de2f97f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm" Nov 02 17:43:41.373811 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.373790 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cbeb6cc-2nzxm\" (UID: \"2783bfae-7a20-40d8-bc9b-52a3de2f97f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm" Nov 02 17:43:41.414285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.414249 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbc8f\" (UniqueName: \"kubernetes.io/projected/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-kube-api-access-hbc8f\") pod \"maas-api-revoke-sub-8cbeb6cc-2nzxm\" (UID: \"2783bfae-7a20-40d8-bc9b-52a3de2f97f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm" Nov 02 17:43:41.464302 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.464251 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 17:43:41.464473 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.464307 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" podUID="0ebc4571-d5c2-45f7-80de-5f79648303f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 17:43:41.469955 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.469926 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:43:41.477116 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.477090 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm" Nov 02 17:43:41.798643 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:41.798609 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm"] Nov 02 17:43:41.799370 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:43:41.799346 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2783bfae_7a20_40d8_bc9b_52a3de2f97f4.slice/crio-c2c9265944e6213bcda03554a70b17c0ca7a37ca957d76df4e3822f2d47425a0 WatchSource:0}: Error finding container c2c9265944e6213bcda03554a70b17c0ca7a37ca957d76df4e3822f2d47425a0: Status 404 returned error can't find the container with id c2c9265944e6213bcda03554a70b17c0ca7a37ca957d76df4e3822f2d47425a0 Nov 02 17:43:42.236283 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:42.236223 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm" event={"ID":"2783bfae-7a20-40d8-bc9b-52a3de2f97f4","Type":"ContainerStarted","Data":"b43ff5722336e269ebea67c8918a2ee1c7bd95a56918a9f3bbc0abd56a668fc1"} Nov 02 17:43:42.236432 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:42.236340 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm" event={"ID":"2783bfae-7a20-40d8-bc9b-52a3de2f97f4","Type":"ContainerStarted","Data":"c2c9265944e6213bcda03554a70b17c0ca7a37ca957d76df4e3822f2d47425a0"} Nov 02 17:43:42.279838 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:42.279777 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm" podStartSLOduration=2.279761663 podStartE2EDuration="2.279761663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:42.277160334 +0000 UTC m=+840.898520111" watchObservedRunningTime="2026-11-02 17:43:42.279761663 +0000 UTC m=+840.901121438" Nov 02 17:43:43.245340 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:43.245304 4860 generic.go:356] "Generic (PLEG): container finished" podID="2783bfae-7a20-40d8-bc9b-52a3de2f97f4" containerID="b43ff5722336e269ebea67c8918a2ee1c7bd95a56918a9f3bbc0abd56a668fc1" exitCode=0 Nov 02 17:43:43.245753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:43.245384 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm" event={"ID":"2783bfae-7a20-40d8-bc9b-52a3de2f97f4","Type":"ContainerDied","Data":"b43ff5722336e269ebea67c8918a2ee1c7bd95a56918a9f3bbc0abd56a668fc1"} Nov 02 17:43:43.303367 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:43.301981 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:43.303367 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:43.303346 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:43.303773 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:43.303754 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 17:43:44.410653 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:44.410627 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm" Nov 02 17:43:44.517369 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:44.517286 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-kube-api-access-hbc8f\" (UniqueName: \"kubernetes.io/projected/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-kube-api-access-hbc8f\") pod \"2783bfae-7a20-40d8-bc9b-52a3de2f97f4\" (UID: \"2783bfae-7a20-40d8-bc9b-52a3de2f97f4\") " Nov 02 17:43:44.517369 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:44.517327 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-maas-api-openshift-ca\") pod \"2783bfae-7a20-40d8-bc9b-52a3de2f97f4\" (UID: \"2783bfae-7a20-40d8-bc9b-52a3de2f97f4\") " Nov 02 17:43:44.517574 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:44.517472 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-maas-api-xks-ca\") pod \"2783bfae-7a20-40d8-bc9b-52a3de2f97f4\" (UID: \"2783bfae-7a20-40d8-bc9b-52a3de2f97f4\") " Nov 02 17:43:44.517721 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:44.517694 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-maas-api-openshift-ca" pod "2783bfae-7a20-40d8-bc9b-52a3de2f97f4" (UID: "2783bfae-7a20-40d8-bc9b-52a3de2f97f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:43:44.517862 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:44.517838 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:43:44.519633 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:44.519596 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-maas-api-xks-ca" pod "2783bfae-7a20-40d8-bc9b-52a3de2f97f4" (UID: "2783bfae-7a20-40d8-bc9b-52a3de2f97f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:43:44.520014 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:44.519988 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-kube-api-access-hbc8f" pod "2783bfae-7a20-40d8-bc9b-52a3de2f97f4" (UID: "2783bfae-7a20-40d8-bc9b-52a3de2f97f4"). InnerVolumeSpecName "kube-api-access-hbc8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:44.619417 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:44.619382 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:43:44.619417 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:44.619418 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbc8f\" (UniqueName: \"kubernetes.io/projected/2783bfae-7a20-40d8-bc9b-52a3de2f97f4-kube-api-access-hbc8f\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:43:45.267734 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:45.267701 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm" Nov 02 17:43:45.267923 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:45.267727 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm" event={"ID":"2783bfae-7a20-40d8-bc9b-52a3de2f97f4","Type":"ContainerDied","Data":"c2c9265944e6213bcda03554a70b17c0ca7a37ca957d76df4e3822f2d47425a0"} Nov 02 17:43:45.267923 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:45.267803 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2c9265944e6213bcda03554a70b17c0ca7a37ca957d76df4e3822f2d47425a0" Nov 02 17:43:51.771945 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:43:51.771907 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:43:52.003460 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:52.003427 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7"] Nov 02 17:43:52.004094 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:52.004079 4860 cpu_manager.go:414] "containerMap: removing container" podUID="2783bfae-7a20-40d8-bc9b-52a3de2f97f4" containerName="revoke-keys" Nov 02 17:43:52.004146 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:52.004095 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2783bfae-7a20-40d8-bc9b-52a3de2f97f4" containerName="revoke-keys" Nov 02 17:43:52.004216 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:52.004207 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2783bfae-7a20-40d8-bc9b-52a3de2f97f4" containerName="revoke-keys" Nov 02 17:43:52.009343 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:52.009322 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7" Nov 02 17:43:52.026281 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:52.026216 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:43:52.038083 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:52.038054 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:43:52.044583 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:52.044556 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7"] Nov 02 17:43:52.124257 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:52.124219 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fb1262-f113-462f-a0ed-fba26c773f2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cc5ad5e-hdbw7\" (UID: \"54fb1262-f113-462f-a0ed-fba26c773f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7" Nov 02 17:43:52.124428 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:52.124284 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj247\" (UniqueName: \"kubernetes.io/projected/54fb1262-f113-462f-a0ed-fba26c773f2e-kube-api-access-mj247\") pod \"maas-api-revoke-sub-3cc5ad5e-hdbw7\" (UID: \"54fb1262-f113-462f-a0ed-fba26c773f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7" Nov 02 17:43:52.124428 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:52.124405 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fb1262-f113-462f-a0ed-fba26c773f2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cc5ad5e-hdbw7\" (UID: \"54fb1262-f113-462f-a0ed-fba26c773f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7" Nov 02 17:43:52.226162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:52.226129 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fb1262-f113-462f-a0ed-fba26c773f2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cc5ad5e-hdbw7\" (UID: \"54fb1262-f113-462f-a0ed-fba26c773f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7" Nov 02 17:43:52.226340 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:52.226203 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj247\" (UniqueName: \"kubernetes.io/projected/54fb1262-f113-462f-a0ed-fba26c773f2e-kube-api-access-mj247\") pod \"maas-api-revoke-sub-3cc5ad5e-hdbw7\" (UID: \"54fb1262-f113-462f-a0ed-fba26c773f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7" Nov 02 17:43:52.226340 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:52.226279 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fb1262-f113-462f-a0ed-fba26c773f2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cc5ad5e-hdbw7\" (UID: \"54fb1262-f113-462f-a0ed-fba26c773f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7" Nov 02 17:43:52.226992 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:52.226966 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fb1262-f113-462f-a0ed-fba26c773f2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cc5ad5e-hdbw7\" (UID: \"54fb1262-f113-462f-a0ed-fba26c773f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7" Nov 02 17:43:52.228554 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:52.228528 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fb1262-f113-462f-a0ed-fba26c773f2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cc5ad5e-hdbw7\" (UID: \"54fb1262-f113-462f-a0ed-fba26c773f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7" Nov 02 17:43:52.250445 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:52.250420 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj247\" (UniqueName: \"kubernetes.io/projected/54fb1262-f113-462f-a0ed-fba26c773f2e-kube-api-access-mj247\") pod \"maas-api-revoke-sub-3cc5ad5e-hdbw7\" (UID: \"54fb1262-f113-462f-a0ed-fba26c773f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7" Nov 02 17:43:52.324635 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:52.324603 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7" Nov 02 17:43:52.639005 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:43:52.638975 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54fb1262_f113_462f_a0ed_fba26c773f2e.slice/crio-196c6e3f967145f2e06d72d3b6ee4fde1e43b7b186c605bf3f596cb311ad9bb7 WatchSource:0}: Error finding container 196c6e3f967145f2e06d72d3b6ee4fde1e43b7b186c605bf3f596cb311ad9bb7: Status 404 returned error can't find the container with id 196c6e3f967145f2e06d72d3b6ee4fde1e43b7b186c605bf3f596cb311ad9bb7 Nov 02 17:43:52.640827 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:52.640796 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7"] Nov 02 17:43:53.338032 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:53.337993 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7" event={"ID":"54fb1262-f113-462f-a0ed-fba26c773f2e","Type":"ContainerStarted","Data":"9bea48887da5b717cdd5bb7c00ccc6901b66552e3c6d8e5efbcdd38d27f5086f"} Nov 02 17:43:53.338446 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:53.338058 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7" event={"ID":"54fb1262-f113-462f-a0ed-fba26c773f2e","Type":"ContainerStarted","Data":"196c6e3f967145f2e06d72d3b6ee4fde1e43b7b186c605bf3f596cb311ad9bb7"} Nov 02 17:43:53.383260 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:53.383204 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7" podStartSLOduration=2.383187995 podStartE2EDuration="2.383187995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:53.380730908 +0000 UTC m=+852.002090683" watchObservedRunningTime="2026-11-02 17:43:53.383187995 +0000 UTC m=+852.004547874" Nov 02 17:43:54.349962 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:54.349918 4860 generic.go:356] "Generic (PLEG): container finished" podID="54fb1262-f113-462f-a0ed-fba26c773f2e" containerID="9bea48887da5b717cdd5bb7c00ccc6901b66552e3c6d8e5efbcdd38d27f5086f" exitCode=0 Nov 02 17:43:54.350397 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:54.350003 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7" event={"ID":"54fb1262-f113-462f-a0ed-fba26c773f2e","Type":"ContainerDied","Data":"9bea48887da5b717cdd5bb7c00ccc6901b66552e3c6d8e5efbcdd38d27f5086f"} Nov 02 17:43:55.514977 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:55.514946 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7" Nov 02 17:43:55.580375 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:55.580346 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54fb1262-f113-462f-a0ed-fba26c773f2e-kube-api-access-mj247\" (UniqueName: \"kubernetes.io/projected/54fb1262-f113-462f-a0ed-fba26c773f2e-kube-api-access-mj247\") pod \"54fb1262-f113-462f-a0ed-fba26c773f2e\" (UID: \"54fb1262-f113-462f-a0ed-fba26c773f2e\") " Nov 02 17:43:55.580556 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:55.580496 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54fb1262-f113-462f-a0ed-fba26c773f2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fb1262-f113-462f-a0ed-fba26c773f2e-maas-api-xks-ca\") pod \"54fb1262-f113-462f-a0ed-fba26c773f2e\" (UID: \"54fb1262-f113-462f-a0ed-fba26c773f2e\") " Nov 02 17:43:55.580637 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:55.580621 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54fb1262-f113-462f-a0ed-fba26c773f2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fb1262-f113-462f-a0ed-fba26c773f2e-maas-api-openshift-ca\") pod \"54fb1262-f113-462f-a0ed-fba26c773f2e\" (UID: \"54fb1262-f113-462f-a0ed-fba26c773f2e\") " Nov 02 17:43:55.581090 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:55.581038 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54fb1262-f113-462f-a0ed-fba26c773f2e-maas-api-openshift-ca" pod "54fb1262-f113-462f-a0ed-fba26c773f2e" (UID: "54fb1262-f113-462f-a0ed-fba26c773f2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:43:55.581226 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:55.581182 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fb1262-f113-462f-a0ed-fba26c773f2e-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:43:55.582628 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:55.582605 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54fb1262-f113-462f-a0ed-fba26c773f2e-maas-api-xks-ca" pod "54fb1262-f113-462f-a0ed-fba26c773f2e" (UID: "54fb1262-f113-462f-a0ed-fba26c773f2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:43:55.582742 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:55.582650 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fb1262-f113-462f-a0ed-fba26c773f2e-kube-api-access-mj247" pod "54fb1262-f113-462f-a0ed-fba26c773f2e" (UID: "54fb1262-f113-462f-a0ed-fba26c773f2e"). InnerVolumeSpecName "kube-api-access-mj247". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:55.683078 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:55.682997 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj247\" (UniqueName: \"kubernetes.io/projected/54fb1262-f113-462f-a0ed-fba26c773f2e-kube-api-access-mj247\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:43:55.683078 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:55.683031 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fb1262-f113-462f-a0ed-fba26c773f2e-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:43:56.368481 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:56.368440 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7" Nov 02 17:43:56.368481 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:56.368467 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7" event={"ID":"54fb1262-f113-462f-a0ed-fba26c773f2e","Type":"ContainerDied","Data":"196c6e3f967145f2e06d72d3b6ee4fde1e43b7b186c605bf3f596cb311ad9bb7"} Nov 02 17:43:56.368786 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:43:56.368516 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="196c6e3f967145f2e06d72d3b6ee4fde1e43b7b186c605bf3f596cb311ad9bb7" Nov 02 17:44:01.327842 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.327809 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9"] Nov 02 17:44:01.328767 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.328744 4860 cpu_manager.go:414] "containerMap: removing container" podUID="54fb1262-f113-462f-a0ed-fba26c773f2e" containerName="revoke-keys" Nov 02 17:44:01.328767 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.328766 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54fb1262-f113-462f-a0ed-fba26c773f2e" containerName="revoke-keys" Nov 02 17:44:01.328975 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.328946 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="54fb1262-f113-462f-a0ed-fba26c773f2e" containerName="revoke-keys" Nov 02 17:44:01.332988 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.332967 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.340582 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.340560 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6"] Nov 02 17:44:01.345891 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.345872 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.347468 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.347444 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc"] Nov 02 17:44:01.352037 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.352018 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.380187 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.380155 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9"] Nov 02 17:44:01.396896 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.396836 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc"] Nov 02 17:44:01.404974 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.404937 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6"] Nov 02 17:44:01.411099 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.411064 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p"] Nov 02 17:44:01.415698 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.415669 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.420040 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.420012 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b"] Nov 02 17:44:01.432111 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.432083 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.455144 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p"] Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457072 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457104 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457120 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457136 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457161 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8568f684-1f8d-444f-854a-5e54b0484861-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457201 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457215 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457233 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8332d1-5646-4263-957d-58fa676e8775-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457248 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457271 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd25476-7eee-41c9-908c-29c04c64f956-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457305 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8568f684-1f8d-444f-854a-5e54b0484861-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457324 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457347 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9gkl\" (UniqueName: \"kubernetes.io/projected/8bd25476-7eee-41c9-908c-29c04c64f956-kube-api-access-g9gkl\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457371 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457395 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdjg6\" (UniqueName: \"kubernetes.io/projected/8568f684-1f8d-444f-854a-5e54b0484861-kube-api-access-qdjg6\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457422 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8568f684-1f8d-444f-854a-5e54b0484861-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457455 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8332d1-5646-4263-957d-58fa676e8775-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457495 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457517 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd25476-7eee-41c9-908c-29c04c64f956-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457558 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd25476-7eee-41c9-908c-29c04c64f956-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457582 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457601 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457616 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmbwd\" (UniqueName: \"kubernetes.io/projected/df8332d1-5646-4263-957d-58fa676e8775-kube-api-access-kmbwd\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457674 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457692 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457708 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.462893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.457741 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8332d1-5646-4263-957d-58fa676e8775-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.507591 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.507556 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b"] Nov 02 17:44:01.558797 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.558764 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.559030 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.558820 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.559030 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.558839 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.559030 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.558898 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8332d1-5646-4263-957d-58fa676e8775-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.559030 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.558928 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.559030 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.558951 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkzzn\" (UniqueName: \"kubernetes.io/projected/20b1a14b-2879-4835-acac-cfa13510175f-kube-api-access-bkzzn\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.559030 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.558978 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b1a14b-2879-4835-acac-cfa13510175f-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.559402 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559109 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.559402 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559173 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b1a14b-2879-4835-acac-cfa13510175f-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.559402 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559204 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.559402 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559262 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.559402 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559289 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.559402 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559292 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdjg6\" (UniqueName: \"kubernetes.io/projected/8568f684-1f8d-444f-854a-5e54b0484861-kube-api-access-qdjg6\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.559402 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559370 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8568f684-1f8d-444f-854a-5e54b0484861-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.559402 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559401 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.559866 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559433 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.559866 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559547 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.559866 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559560 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.559866 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559581 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.559866 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559657 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd25476-7eee-41c9-908c-29c04c64f956-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.559866 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559719 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndjsx\" (UniqueName: \"kubernetes.io/projected/48dce99f-6608-4c61-b296-2387ff5d2caa-kube-api-access-ndjsx\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.559866 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559758 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmbwd\" (UniqueName: \"kubernetes.io/projected/df8332d1-5646-4263-957d-58fa676e8775-kube-api-access-kmbwd\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.559866 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559823 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559871 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559907 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559924 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8568f684-1f8d-444f-854a-5e54b0484861-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559937 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559949 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dce99f-6608-4c61-b296-2387ff5d2caa-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.559986 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8332d1-5646-4263-957d-58fa676e8775-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.560009 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.560059 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.560085 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.560104 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd25476-7eee-41c9-908c-29c04c64f956-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.560129 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8568f684-1f8d-444f-854a-5e54b0484861-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.560157 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.560179 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.560205 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9gkl\" (UniqueName: \"kubernetes.io/projected/8bd25476-7eee-41c9-908c-29c04c64f956-kube-api-access-g9gkl\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.560239 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.560249 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.560306 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8332d1-5646-4263-957d-58fa676e8775-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.560357 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd25476-7eee-41c9-908c-29c04c64f956-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.560403 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.560408 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd25476-7eee-41c9-908c-29c04c64f956-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.560423 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b1a14b-2879-4835-acac-cfa13510175f-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.560483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.560469 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.561611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.560774 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.561611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.560951 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.561611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.561052 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.561611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.561354 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.561830 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.561611 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.561830 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.561696 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.561830 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.561723 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.561830 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.561766 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.562123 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.561964 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.562208 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.562186 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.562269 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.562256 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.562326 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.562306 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.562577 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.562556 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.563274 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.562845 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.563533 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.563471 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8332d1-5646-4263-957d-58fa676e8775-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.563689 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.563669 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.563747 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.563733 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8568f684-1f8d-444f-854a-5e54b0484861-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.563840 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.563818 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd25476-7eee-41c9-908c-29c04c64f956-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.563926 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.563879 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8568f684-1f8d-444f-854a-5e54b0484861-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.564608 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.564585 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.564821 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.564802 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8332d1-5646-4263-957d-58fa676e8775-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.583522 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.583469 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8332d1-5646-4263-957d-58fa676e8775-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.584156 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.584134 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdjg6\" (UniqueName: \"kubernetes.io/projected/8568f684-1f8d-444f-854a-5e54b0484861-kube-api-access-qdjg6\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.585542 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.585519 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8568f684-1f8d-444f-854a-5e54b0484861-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.585952 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.585922 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd25476-7eee-41c9-908c-29c04c64f956-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.586400 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.586382 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmbwd\" (UniqueName: \"kubernetes.io/projected/df8332d1-5646-4263-957d-58fa676e8775-kube-api-access-kmbwd\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.604647 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.604619 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9gkl\" (UniqueName: \"kubernetes.io/projected/8bd25476-7eee-41c9-908c-29c04c64f956-kube-api-access-g9gkl\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.651511 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.651470 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:01.663844 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.663820 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:01.664469 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.664442 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dce99f-6608-4c61-b296-2387ff5d2caa-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.664546 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.664530 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.664596 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.664582 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.664647 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.664633 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.664748 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.664731 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b1a14b-2879-4835-acac-cfa13510175f-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.664809 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.664771 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.664892 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.664818 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.664955 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.664927 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.665008 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.664984 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.665058 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.665029 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkzzn\" (UniqueName: \"kubernetes.io/projected/20b1a14b-2879-4835-acac-cfa13510175f-kube-api-access-bkzzn\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.665058 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.665053 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b1a14b-2879-4835-acac-cfa13510175f-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.665170 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.665055 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.665170 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.665077 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.665170 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.665144 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b1a14b-2879-4835-acac-cfa13510175f-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.665332 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.665178 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.665332 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.665242 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.665332 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.665276 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.665483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.665339 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.665483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.665391 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndjsx\" (UniqueName: \"kubernetes.io/projected/48dce99f-6608-4c61-b296-2387ff5d2caa-kube-api-access-ndjsx\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.665483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.665397 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.665990 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.665712 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.665990 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.665924 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.665990 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.664987 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.666218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.666078 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.666218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.665244 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dce99f-6608-4c61-b296-2387ff5d2caa-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.666218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.666176 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.666750 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.666724 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b1a14b-2879-4835-acac-cfa13510175f-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.666899 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.666880 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.668408 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.668385 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b1a14b-2879-4835-acac-cfa13510175f-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.668595 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.668575 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.668716 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.668695 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.668939 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.668922 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.669664 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.669641 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:01.696611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.696590 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.696781 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.696760 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndjsx\" (UniqueName: \"kubernetes.io/projected/48dce99f-6608-4c61-b296-2387ff5d2caa-kube-api-access-ndjsx\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.698231 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.698213 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b1a14b-2879-4835-acac-cfa13510175f-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.698495 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.698439 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkzzn\" (UniqueName: \"kubernetes.io/projected/20b1a14b-2879-4835-acac-cfa13510175f-kube-api-access-bkzzn\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-2k98b\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.746033 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.745940 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:01.760593 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.760564 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:01.999353 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:01.999323 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9"] Nov 02 17:44:01.999500 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:44:01.999480 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8568f684_1f8d_444f_854a_5e54b0484861.slice/crio-6d5d08cfe410a56630c12751d150eb423e66a6d752be0c6a83c46a39fb2e6f0b WatchSource:0}: Error finding container 6d5d08cfe410a56630c12751d150eb423e66a6d752be0c6a83c46a39fb2e6f0b: Status 404 returned error can't find the container with id 6d5d08cfe410a56630c12751d150eb423e66a6d752be0c6a83c46a39fb2e6f0b Nov 02 17:44:02.002624 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.002583 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:02.002762 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.002738 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:02.002881 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.002838 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:02.046939 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.046910 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6"] Nov 02 17:44:02.048576 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:44:02.048542 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8332d1_5646_4263_957d_58fa676e8775.slice/crio-1ff20032378e2b02db63aadc1eea1809771c7534228510bf5464c78af3810791 WatchSource:0}: Error finding container 1ff20032378e2b02db63aadc1eea1809771c7534228510bf5464c78af3810791: Status 404 returned error can't find the container with id 1ff20032378e2b02db63aadc1eea1809771c7534228510bf5464c78af3810791 Nov 02 17:44:02.053476 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.053432 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:02.053476 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:44:02.053463 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd25476_7eee_41c9_908c_29c04c64f956.slice/crio-4366358493aba3204c361e73a23c1f49b5c596117e4f67a4a20ae3c70e200c88 WatchSource:0}: Error finding container 4366358493aba3204c361e73a23c1f49b5c596117e4f67a4a20ae3c70e200c88: Status 404 returned error can't find the container with id 4366358493aba3204c361e73a23c1f49b5c596117e4f67a4a20ae3c70e200c88 Nov 02 17:44:02.053622 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.053524 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:02.053622 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.053566 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:02.054244 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.054129 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc"] Nov 02 17:44:02.056352 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.056323 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:02.056476 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.056392 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:02.056476 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.056450 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:02.141993 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.138998 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p"] Nov 02 17:44:02.149904 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.149185 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b"] Nov 02 17:44:02.154104 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.153517 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:02.154104 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.153565 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:02.154104 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.153600 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:02.154104 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.153641 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:02.154104 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.153642 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:02.154104 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.153766 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:02.438704 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.438609 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" event={"ID":"20b1a14b-2879-4835-acac-cfa13510175f","Type":"ContainerStarted","Data":"4945f033c3321035a66c43ef93a2633e49e63f0ec364c4b5dd27fd9d431f979e"} Nov 02 17:44:02.438704 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.438676 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" event={"ID":"20b1a14b-2879-4835-acac-cfa13510175f","Type":"ContainerStarted","Data":"4f1c318afc094c86b6536c8bc07f8678f766e69a23af2785d728119ed0cb6f16"} Nov 02 17:44:02.442309 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.442275 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" event={"ID":"8568f684-1f8d-444f-854a-5e54b0484861","Type":"ContainerStarted","Data":"761ba773b36c43ac72ea1f952ef31299046ae47d74a186f0e80f6c5a5c9220dd"} Nov 02 17:44:02.442434 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.442334 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" event={"ID":"8568f684-1f8d-444f-854a-5e54b0484861","Type":"ContainerStarted","Data":"6d5d08cfe410a56630c12751d150eb423e66a6d752be0c6a83c46a39fb2e6f0b"} Nov 02 17:44:02.445239 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.445208 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" event={"ID":"8bd25476-7eee-41c9-908c-29c04c64f956","Type":"ContainerStarted","Data":"1b307520b0bd5961ea1b07b32a774511c62adcc9ae774110d5dcd3c3cb6ebc83"} Nov 02 17:44:02.445334 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.445261 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" event={"ID":"8bd25476-7eee-41c9-908c-29c04c64f956","Type":"ContainerStarted","Data":"4366358493aba3204c361e73a23c1f49b5c596117e4f67a4a20ae3c70e200c88"} Nov 02 17:44:02.448349 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.448323 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" event={"ID":"df8332d1-5646-4263-957d-58fa676e8775","Type":"ContainerStarted","Data":"23a0702d9b64b96ffbd9117bd626910e7693754b970d9ef1a66b6126884c7d15"} Nov 02 17:44:02.448432 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.448370 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" event={"ID":"df8332d1-5646-4263-957d-58fa676e8775","Type":"ContainerStarted","Data":"1ff20032378e2b02db63aadc1eea1809771c7534228510bf5464c78af3810791"} Nov 02 17:44:02.451864 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.450877 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" event={"ID":"48dce99f-6608-4c61-b296-2387ff5d2caa","Type":"ContainerStarted","Data":"c282f97739a44629edc3ae9cd598928f8f13e61f1231f7a88b930130e13c5cc3"} Nov 02 17:44:02.451864 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.450921 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" event={"ID":"48dce99f-6608-4c61-b296-2387ff5d2caa","Type":"ContainerStarted","Data":"592c25c66b303743120d902c908618f73d739c70f819d9c80cdd04c22a051821"} Nov 02 17:44:02.511635 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.511581 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" podStartSLOduration=1.5115614050000001 podStartE2EDuration="1.511561405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:02.495351166 +0000 UTC m=+861.116710951" watchObservedRunningTime="2026-11-02 17:44:02.511561405 +0000 UTC m=+861.132921180" Nov 02 17:44:02.571831 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.569637 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" podStartSLOduration=1.5696190159999999 podStartE2EDuration="1.569619016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:02.563409703 +0000 UTC m=+861.184769502" watchObservedRunningTime="2026-11-02 17:44:02.569619016 +0000 UTC m=+861.190978792" Nov 02 17:44:02.651731 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.651700 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:02.664968 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.664936 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:02.670112 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.670090 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:02.682819 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.681873 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" podStartSLOduration=1.681839723 podStartE2EDuration="1.681839723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:02.677970724 +0000 UTC m=+861.299330501" watchObservedRunningTime="2026-11-02 17:44:02.681839723 +0000 UTC m=+861.303199501" Nov 02 17:44:02.702926 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.689089 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" podStartSLOduration=1.689031685 podStartE2EDuration="1.689031685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:02.625083903 +0000 UTC m=+861.246443683" watchObservedRunningTime="2026-11-02 17:44:02.689031685 +0000 UTC m=+861.310391471" Nov 02 17:44:02.750491 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.750417 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" podStartSLOduration=1.7503919159999999 podStartE2EDuration="1.750391916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:02.739585229 +0000 UTC m=+861.360945007" watchObservedRunningTime="2026-11-02 17:44:02.750391916 +0000 UTC m=+861.371751696" Nov 02 17:44:02.753084 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.753001 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:02.761030 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:02.761000 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:03.670875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:03.661128 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:44:03.670875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:03.661191 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 17:44:03.696872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:03.696632 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 17:44:03.696872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:03.696688 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 17:44:03.696872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:03.696769 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:44:03.696872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:03.696791 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" podUID="df8332d1-5646-4263-957d-58fa676e8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 17:44:03.779880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:03.767977 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:44:03.779880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:03.768038 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 17:44:03.779880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:03.768125 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 17:44:03.779880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:03.768147 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 17:44:04.336136 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.334961 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:04.345717 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.344928 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:04.461894 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.458493 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 02 17:44:04.567875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.565977 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:04.658484 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.658361 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:04.659155 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.659111 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:44:04.659320 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.659297 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 17:44:04.676008 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.675972 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 17:44:04.689204 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.689152 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 17:44:04.689434 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.688935 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:44:04.689615 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.689593 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" podUID="df8332d1-5646-4263-957d-58fa676e8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 17:44:04.717873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.717649 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:04.742875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.737629 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:04.742875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.739526 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:04.767876 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.757747 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 17:44:04.767876 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.757801 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 17:44:04.771871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.768244 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:44:04.771871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.768300 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 17:44:04.883874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.876929 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 02 17:44:04.900966 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.897358 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:04.943932 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:04.924931 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:05.001874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.001623 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded" Nov 02 17:44:05.067875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.066981 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:05.067875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.067451 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 02 17:44:05.090875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.068587 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:05.106874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.105977 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:05.338178 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.338132 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 02 17:44:05.338926 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.338901 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:44:05.349070 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.348945 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:05.349723 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.349699 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:44:05.470064 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.470015 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:05.493215 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.471052 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:44:05.654801 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.654713 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:44:05.655105 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.655055 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 17:44:05.666986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.666730 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:44:05.666986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.666781 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" podUID="df8332d1-5646-4263-957d-58fa676e8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 17:44:05.666986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.666899 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:05.752414 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.740109 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:05.752414 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.740991 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 17:44:05.752414 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.741191 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded" Nov 02 17:44:05.792182 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.788395 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 17:44:05.792182 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.788484 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 17:44:05.826913 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.819036 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 17:44:05.826913 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.819117 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 17:44:05.826913 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.821247 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:44:05.826913 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.821294 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 17:44:05.897275 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.880015 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:05.898512 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.898472 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:05.998934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:05.959314 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:06.067987 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.041366 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:44:06.068807 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.068785 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:44:06.078770 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.069046 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:44:06.079025 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.068689 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:06.079173 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.068057 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:06.182677 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.068195 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:06.182924 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.109972 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:06.196877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.116489 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:44:06.196877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.184049 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:44:06.196877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.184093 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:44:06.196877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.184130 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:44:06.196877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.184166 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:44:06.196877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.184201 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:44:06.196877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.184237 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:44:06.408936 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.372383 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 02 17:44:06.465975 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.399953 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:06.618177 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.470946 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:06.696873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.693296 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 02 17:44:06.731873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.693786 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 17:44:06.731873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.731756 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 17:44:06.753806 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.734838 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bvtck" podUID="34275f86-46f6-4b1d-bd0e-865383413fcf" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.48:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:06.793124 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.793071 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 02 17:44:06.901966 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.841663 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:44:06.901966 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.892243 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 17:44:06.901966 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.841731 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 17:44:06.901966 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.892299 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 17:44:06.901966 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.867275 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:44:06.901966 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.892341 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" podUID="df8332d1-5646-4263-957d-58fa676e8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 17:44:06.901966 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.882248 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:44:06.901966 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.892376 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 17:44:06.901966 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.867766 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:06.901966 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.893345 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:06.937674 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.937623 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:06.939214 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.939185 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 17:44:06.980485 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:06.973917 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:07.127876 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.126310 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:07.172923 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.150504 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded" Nov 02 17:44:07.199873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.198796 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 02 17:44:07.199873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.199703 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 02 17:44:07.207884 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.200445 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 17:44:07.493930 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.490897 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:07.628465 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.628421 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:07.697873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.680953 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:44:07.697873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.681018 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" podUID="df8332d1-5646-4263-957d-58fa676e8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 17:44:07.697873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.681892 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 17:44:07.697873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.681938 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 17:44:07.698388 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.698351 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:07.723876 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.718053 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:07.723876 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.718550 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:44:07.723876 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.718594 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 17:44:07.813966 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.768281 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 17:44:07.813966 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.768351 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 17:44:07.813966 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.769255 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:44:07.813966 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.769305 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 17:44:07.934915 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.920782 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:07.934915 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.921187 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 02 17:44:07.985908 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:07.938968 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:08.057905 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.017522 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:08.086061 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.085958 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 02 17:44:08.278875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.215808 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:08.314876 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.241798 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:08.315460 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.279394 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:08.360915 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.360795 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 02 17:44:08.619931 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.599133 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:08.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.610634 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Nov 02 17:44:08.648595 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.648540 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:08.673222 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.654207 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:44:08.673539 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.673489 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 17:44:08.673692 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.664338 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:44:08.673830 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.673811 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" podUID="df8332d1-5646-4263-957d-58fa676e8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 17:44:08.673953 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.671211 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 17:44:08.674096 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.674075 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 17:44:08.711061 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.710610 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:08.723759 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.722937 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:08.757789 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.757669 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 17:44:08.757789 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.757725 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 17:44:08.767889 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.767738 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:44:08.767889 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.767795 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 17:44:08.948812 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.945959 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 02 17:44:08.948812 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.946382 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 02 17:44:08.948812 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:08.948459 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 02 17:44:09.113875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:09.041594 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 17:44:09.126019 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:09.086400 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:09.340537 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:09.340494 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:09.403714 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:09.345005 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 17:44:09.443134 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:09.304440 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 02 17:44:09.639930 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:09.514315 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Nov 02 17:44:09.742312 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:09.643709 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 02 17:44:09.998968 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:09.680231 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:44:10.000003 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:09.729960 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 02 17:44:10.013911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:09.730473 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 02 17:44:10.013911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:09.730792 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:44:10.013911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.010272 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 17:44:10.013911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:09.947430 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:10.013911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:09.756165 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 02 17:44:10.013911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:09.756126 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 17:44:10.013911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.010803 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 17:44:10.013911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:09.795602 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:44:10.013911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.010866 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 17:44:10.013911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:09.796279 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:44:10.013911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.010904 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" podUID="df8332d1-5646-4263-957d-58fa676e8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 17:44:10.013911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:09.796320 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 17:44:10.013911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.010938 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 17:44:10.013911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:09.947464 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 02 17:44:10.013911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:09.948586 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:10.128899 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.115994 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:10.163880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.148452 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:10.415498 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.414939 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:10.458881 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.446391 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:10.508278 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.494525 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:44:10.674212 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.674125 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:44:10.674746 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.674714 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 17:44:10.717740 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.715103 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:44:10.717740 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.715166 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" podUID="df8332d1-5646-4263-957d-58fa676e8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 17:44:10.717740 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.715255 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 17:44:10.717740 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.715276 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 17:44:10.753386 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.753326 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:10.788090 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.788033 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:10.788410 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.788389 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded" Nov 02 17:44:10.790495 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.789950 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:44:10.790495 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.790001 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 17:44:10.790689 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.790517 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 17:44:10.790689 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.790556 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 17:44:10.991788 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:10.991692 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 02 17:44:11.065123 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.009056 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:11.091045 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.009978 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 02 17:44:11.092456 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.011068 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:11.101403 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.011098 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 02 17:44:11.109563 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.011118 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 02 17:44:11.126057 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.012403 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:11.144013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.120173 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:11.149992 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.149951 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:11.436026 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.435983 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 02 17:44:11.465992 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.465951 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 17:44:11.571615 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.571584 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:44:11.662466 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.653832 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:44:11.662883 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.662689 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 17:44:11.667967 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.667939 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:44:11.668139 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.668113 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" podUID="df8332d1-5646-4263-957d-58fa676e8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 17:44:11.683879 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.681967 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 17:44:11.683879 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.682099 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 17:44:11.772648 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.772560 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Nov 02 17:44:11.773004 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.772979 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 17:44:11.773152 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.773131 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 17:44:11.797757 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.797716 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 02 17:44:11.814252 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.814222 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:44:11.821543 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.821510 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:44:11.841869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:11.822237 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 17:44:12.185914 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.172971 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 02 17:44:12.193887 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.173927 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 17:44:12.193887 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.173968 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 02 17:44:12.193887 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.174001 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:12.193887 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.189023 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 02 17:44:12.193887 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.189501 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:12.193887 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.188944 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:12.193887 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.189699 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:12.478182 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.478059 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 02 17:44:12.495650 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.495605 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:12.718753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.718713 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:44:12.719056 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.719019 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 17:44:12.719301 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.719280 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:44:12.719428 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.719411 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" podUID="df8332d1-5646-4263-957d-58fa676e8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 17:44:12.730094 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.729995 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 17:44:12.730341 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.730311 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 17:44:12.759872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.758540 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 17:44:12.759872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.758614 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 17:44:12.761121 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.761092 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:44:12.761599 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.761553 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 17:44:12.799091 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.799052 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:12.929882 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:12.928981 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 02 17:44:13.196184 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.196151 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:13.197205 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.197167 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 02 17:44:13.199284 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.199246 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:13.213763 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.213714 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:13.214740 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.214706 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:13.215603 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.215579 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 02 17:44:13.229142 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.196088 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:13.230103 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.230069 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:13.488929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.487981 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 02 17:44:13.488929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.488038 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:13.528978 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.528931 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:13.678882 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.675335 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 17:44:13.678882 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.675395 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 17:44:13.678882 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.676147 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:44:13.678882 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.676188 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 17:44:13.678882 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.677691 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:44:13.678882 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.677729 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" podUID="df8332d1-5646-4263-957d-58fa676e8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 17:44:13.780028 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.779945 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:44:13.780290 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.780258 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 17:44:13.801555 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.800927 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:13.931084 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.931033 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:13.931914 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:13.931886 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:44:14.017540 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.017500 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:14.197991 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.197943 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:14.198935 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.198195 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:14.200295 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.199589 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 02 17:44:14.220491 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.220454 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 02 17:44:14.222348 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.221024 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 02 17:44:14.222348 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.221307 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:14.245779 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.244514 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:14.322873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.321954 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 02 17:44:14.593003 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.592949 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:14.623873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.593574 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:14.624463 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.607961 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:44:14.671969 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.626042 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 17:44:14.710873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.630331 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:14.715881 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.677994 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 17:44:14.798965 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.798921 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 17:44:14.799242 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.682965 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:44:14.799376 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.799353 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" podUID="df8332d1-5646-4263-957d-58fa676e8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 17:44:14.805300 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.805238 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 17:44:14.805477 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.805453 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 17:44:14.827987 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.815404 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Nov 02 17:44:14.827987 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.815512 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:14.861399 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.860945 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:14.861399 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.861012 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:14.935878 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:14.934407 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:15.030731 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:15.030687 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:15.247922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:15.201952 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:15.247922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:15.202756 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 02 17:44:15.247922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:15.204649 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:15.247922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:15.220961 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 02 17:44:15.247922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:15.222249 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:15.247922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:15.222550 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 02 17:44:15.251388 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:15.249719 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:15.324984 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:15.324941 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 02 17:44:15.445971 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:15.445937 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:44:15.697878 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:15.620287 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 02 17:44:15.715874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:15.646014 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 02 17:44:15.749874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:15.732672 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:15.749874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:15.734231 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:15.776873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:15.761045 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:15.776873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:15.761112 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:16.109872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.088051 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:44:16.113963 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.113927 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 02 17:44:16.186619 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.186581 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:16.236275 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.216954 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:16.236275 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.217359 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 02 17:44:16.239337 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.239292 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 17:44:16.240097 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.239957 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:16.240272 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.240248 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 02 17:44:16.306763 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.306716 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 02 17:44:16.318979 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.318944 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:16.319209 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.319181 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:16.345868 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.341638 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:16.651871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.650817 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 02 17:44:16.680874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.668025 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:16.709584 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.708956 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:16.740874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.714911 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:16.741223 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.741198 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:16.758125 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.758077 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:16.795934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.781373 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:16.796222 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.796175 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" podUID="df8332d1-5646-4263-957d-58fa676e8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:16.797130 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.783927 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 17:44:16.871027 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.870975 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:16.936999 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:16.933184 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:17.127091 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.127045 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:17.127385 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.127348 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:17.167874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.156011 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:17.167874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.156070 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:17.183737 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.183610 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l"] Nov 02 17:44:17.238957 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.207021 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:17.238957 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.220958 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:17.238957 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.221136 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:17.240839 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.240794 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded" Nov 02 17:44:17.270118 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.270073 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 02 17:44:17.305996 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.270083 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 17:44:17.306914 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.306879 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:17.316344 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.291225 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:17.338245 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.300907 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:17.345315 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.321442 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": context deadline exceeded" Nov 02 17:44:17.345925 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.270192 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:17.375831 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.375780 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:17.419841 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.417558 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 02 17:44:17.420695 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.417811 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:17.420927 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.420897 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:17.463887 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.454800 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:17.507650 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.496897 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l"] Nov 02 17:44:17.906358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.882358 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:17.906358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.882645 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" podUID="df8332d1-5646-4263-957d-58fa676e8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:17.930396 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.930311 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:17.930735 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.930709 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:17.931102 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.931072 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:17.931276 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.931248 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:17.931476 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.931459 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:17.931657 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.931626 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7dwp\" (UniqueName: \"kubernetes.io/projected/64e985f1-f451-4ee1-ba69-76ce8949ca40-kube-api-access-c7dwp\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:17.970630 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:17.970494 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.077368 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.077323 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.077811 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.077791 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e985f1-f451-4ee1-ba69-76ce8949ca40-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.102088 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.084285 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:18.102273 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.084310 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 02 17:44:18.105489 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.097930 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:18.105489 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.097957 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:18.109168 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.109112 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:44:18.109411 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.109393 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:44:18.109817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.109798 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:18.157518 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.155932 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:18.157518 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.155997 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:18.168869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.161931 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:18.168869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.161988 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:18.180088 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.180059 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 17:44:18.211143 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.211096 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:18.211607 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.211592 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:44:18.225431 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.225399 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.225662 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.225645 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.225907 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.225892 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.226033 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.226019 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.226144 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.226125 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.226253 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.226239 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7dwp\" (UniqueName: \"kubernetes.io/projected/64e985f1-f451-4ee1-ba69-76ce8949ca40-kube-api-access-c7dwp\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.226392 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.226377 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.226484 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.226472 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.226763 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.226750 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e985f1-f451-4ee1-ba69-76ce8949ca40-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.240052 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.227663 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e985f1-f451-4ee1-ba69-76ce8949ca40-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.290444 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.241650 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.290685 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.248931 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:18.290871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.290826 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:18.301177 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.276906 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:18.326076 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.322427 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:18.359083 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.322459 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 02 17:44:18.359602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.334981 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:18.430929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.330924 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:18.430929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.372931 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:18.477477 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.475338 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.477477 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.476282 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.477477 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.476527 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.486907 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.478968 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:18.505871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.505813 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 02 17:44:18.602821 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.602789 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.643973 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.625188 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:18.643973 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.625249 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:18.669872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.650484 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.671825 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.671801 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:44:18.701929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.688965 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Nov 02 17:44:18.725177 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.725150 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:44:18.734906 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.732519 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:44:18.734906 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.733981 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:44:18.734906 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.734775 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:44:18.737202 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.737157 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:44:18.743506 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.743487 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:44:18.760486 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.750371 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:44:18.786110 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.761115 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7dwp\" (UniqueName: \"kubernetes.io/projected/64e985f1-f451-4ee1-ba69-76ce8949ca40-kube-api-access-c7dwp\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:18.979930 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:18.977180 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:44:19.130940 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:19.130021 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:19.220537 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:19.220507 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:44:19.306638 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:19.284362 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:19.307998 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:19.307965 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:19.308269 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:19.308222 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:19.344875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:19.342383 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:19.344875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:19.344753 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:44:19.353875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:19.353019 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:19.353875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:19.353069 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:19.369879 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:19.368053 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:19.384882 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:19.383990 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:19.501969 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:19.497001 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:44:19.571949 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:19.505823 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:44:19.571949 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:19.507417 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:19.631980 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:19.631946 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:44:19.720900 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:19.703963 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb"] Nov 02 17:44:19.720900 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:19.706715 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" podUID="6d6284d8-6036-4088-a7e0-a1d902f27ecb" containerName="istio-proxy" containerID="cri-o://589d2ccd71935caf23ff4b0c79740268505c3699ba457eca5c3fa94cb67c335c" gracePeriod=30 Nov 02 17:44:20.005133 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.004843 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:20.006324 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.006301 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:20.008000 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.007977 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:44:20.026721 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.026685 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:44:20.075987 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.075956 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v"] Nov 02 17:44:20.081636 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.081613 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v" Nov 02 17:44:20.111871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.108729 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:44:20.111871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.108926 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:44:20.111871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.111512 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v"] Nov 02 17:44:20.142031 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:44:20.140316 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e985f1_f451_4ee1_ba69_76ce8949ca40.slice/crio-045f6e4591981038557a2faf47c1ce35b57d1087429383a6a2f5a463d306fc8d WatchSource:0}: Error finding container 045f6e4591981038557a2faf47c1ce35b57d1087429383a6a2f5a463d306fc8d: Status 404 returned error can't find the container with id 045f6e4591981038557a2faf47c1ce35b57d1087429383a6a2f5a463d306fc8d Nov 02 17:44:20.175193 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.175168 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l"] Nov 02 17:44:20.175399 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.175278 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:20.175569 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.175546 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:20.175795 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.175765 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:20.181724 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.181688 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99zq8\" (UniqueName: \"kubernetes.io/projected/cd9777f9-d834-4e07-8493-6bd7f392be05-kube-api-access-99zq8\") pod \"maas-api-revoke-sub-bc7eba10-lj86v\" (UID: \"cd9777f9-d834-4e07-8493-6bd7f392be05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v" Nov 02 17:44:20.181918 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.181892 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9777f9-d834-4e07-8493-6bd7f392be05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc7eba10-lj86v\" (UID: \"cd9777f9-d834-4e07-8493-6bd7f392be05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v" Nov 02 17:44:20.182067 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.182046 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9777f9-d834-4e07-8493-6bd7f392be05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc7eba10-lj86v\" (UID: \"cd9777f9-d834-4e07-8493-6bd7f392be05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v" Nov 02 17:44:20.284415 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.284148 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99zq8\" (UniqueName: \"kubernetes.io/projected/cd9777f9-d834-4e07-8493-6bd7f392be05-kube-api-access-99zq8\") pod \"maas-api-revoke-sub-bc7eba10-lj86v\" (UID: \"cd9777f9-d834-4e07-8493-6bd7f392be05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v" Nov 02 17:44:20.284415 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.284184 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9777f9-d834-4e07-8493-6bd7f392be05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc7eba10-lj86v\" (UID: \"cd9777f9-d834-4e07-8493-6bd7f392be05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v" Nov 02 17:44:20.284415 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.284233 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9777f9-d834-4e07-8493-6bd7f392be05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc7eba10-lj86v\" (UID: \"cd9777f9-d834-4e07-8493-6bd7f392be05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v" Nov 02 17:44:20.285170 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.285138 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9777f9-d834-4e07-8493-6bd7f392be05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc7eba10-lj86v\" (UID: \"cd9777f9-d834-4e07-8493-6bd7f392be05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v" Nov 02 17:44:20.286704 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.286680 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9777f9-d834-4e07-8493-6bd7f392be05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc7eba10-lj86v\" (UID: \"cd9777f9-d834-4e07-8493-6bd7f392be05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v" Nov 02 17:44:20.309686 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.309647 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99zq8\" (UniqueName: \"kubernetes.io/projected/cd9777f9-d834-4e07-8493-6bd7f392be05-kube-api-access-99zq8\") pod \"maas-api-revoke-sub-bc7eba10-lj86v\" (UID: \"cd9777f9-d834-4e07-8493-6bd7f392be05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v" Nov 02 17:44:20.400818 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.400730 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v" Nov 02 17:44:20.458093 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.458054 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" event={"ID":"64e985f1-f451-4ee1-ba69-76ce8949ca40","Type":"ContainerStarted","Data":"9ab0311433b9c038b398b73f30b5f6c7f2ca8f3a5a69d1fe911e59a954a72c97"} Nov 02 17:44:20.458329 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.458122 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" event={"ID":"64e985f1-f451-4ee1-ba69-76ce8949ca40","Type":"ContainerStarted","Data":"045f6e4591981038557a2faf47c1ce35b57d1087429383a6a2f5a463d306fc8d"} Nov 02 17:44:20.518720 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.518656 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" podStartSLOduration=4.518640265 podStartE2EDuration="4.518640265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:20.510944444 +0000 UTC m=+879.132304258" watchObservedRunningTime="2026-11-02 17:44:20.518640265 +0000 UTC m=+879.140000040" Nov 02 17:44:20.761742 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:44:20.761710 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9777f9_d834_4e07_8493_6bd7f392be05.slice/crio-0b597a388bdc003f13a90ab162f3e3682759d9e5b5ace073b719594b58a743e8 WatchSource:0}: Error finding container 0b597a388bdc003f13a90ab162f3e3682759d9e5b5ace073b719594b58a743e8: Status 404 returned error can't find the container with id 0b597a388bdc003f13a90ab162f3e3682759d9e5b5ace073b719594b58a743e8 Nov 02 17:44:20.763321 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:20.763291 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v"] Nov 02 17:44:21.039055 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.038967 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7"] Nov 02 17:44:21.044778 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.044755 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7" Nov 02 17:44:21.063069 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.063030 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7"] Nov 02 17:44:21.097923 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.097884 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3dc45e8-t4cz7\" (UID: \"80578a9c-ef49-49de-a95c-eb8d9e1a79bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7" Nov 02 17:44:21.098088 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.098070 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3dc45e8-t4cz7\" (UID: \"80578a9c-ef49-49de-a95c-eb8d9e1a79bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7" Nov 02 17:44:21.098183 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.098169 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x7gc\" (UniqueName: \"kubernetes.io/projected/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-kube-api-access-2x7gc\") pod \"maas-api-revoke-sub-a3dc45e8-t4cz7\" (UID: \"80578a9c-ef49-49de-a95c-eb8d9e1a79bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7" Nov 02 17:44:21.205558 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.205520 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3dc45e8-t4cz7\" (UID: \"80578a9c-ef49-49de-a95c-eb8d9e1a79bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7" Nov 02 17:44:21.205868 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.205820 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x7gc\" (UniqueName: \"kubernetes.io/projected/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-kube-api-access-2x7gc\") pod \"maas-api-revoke-sub-a3dc45e8-t4cz7\" (UID: \"80578a9c-ef49-49de-a95c-eb8d9e1a79bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7" Nov 02 17:44:21.206208 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.206183 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3dc45e8-t4cz7\" (UID: \"80578a9c-ef49-49de-a95c-eb8d9e1a79bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7" Nov 02 17:44:21.206454 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.206432 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3dc45e8-t4cz7\" (UID: \"80578a9c-ef49-49de-a95c-eb8d9e1a79bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7" Nov 02 17:44:21.208967 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.208936 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3dc45e8-t4cz7\" (UID: \"80578a9c-ef49-49de-a95c-eb8d9e1a79bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7" Nov 02 17:44:21.235168 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.235139 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x7gc\" (UniqueName: \"kubernetes.io/projected/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-kube-api-access-2x7gc\") pod \"maas-api-revoke-sub-a3dc45e8-t4cz7\" (UID: \"80578a9c-ef49-49de-a95c-eb8d9e1a79bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7" Nov 02 17:44:21.370874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.367161 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7" Nov 02 17:44:21.378122 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.378092 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:21.379603 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.379426 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:44:21.379603 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.379500 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" podUID="64e985f1-f451-4ee1-ba69-76ce8949ca40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 17:44:21.520120 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.520087 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v" event={"ID":"cd9777f9-d834-4e07-8493-6bd7f392be05","Type":"ContainerStarted","Data":"a8156fb909b3efbf0c813157814c2f36614fc35287910d0a8e599bfe041e0567"} Nov 02 17:44:21.520511 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.520132 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v" event={"ID":"cd9777f9-d834-4e07-8493-6bd7f392be05","Type":"ContainerStarted","Data":"0b597a388bdc003f13a90ab162f3e3682759d9e5b5ace073b719594b58a743e8"} Nov 02 17:44:21.585874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.576838 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v" podStartSLOduration=1.576815852 podStartE2EDuration="1.576815852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:21.574421306 +0000 UTC m=+880.195781083" watchObservedRunningTime="2026-11-02 17:44:21.576815852 +0000 UTC m=+880.198175629" Nov 02 17:44:21.722254 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:44:21.722217 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80578a9c_ef49_49de_a95c_eb8d9e1a79bd.slice/crio-add4b6cccc045c0674ef3e63f985a9529af7d894db4815cd943546a58ebb847a WatchSource:0}: Error finding container add4b6cccc045c0674ef3e63f985a9529af7d894db4815cd943546a58ebb847a: Status 404 returned error can't find the container with id add4b6cccc045c0674ef3e63f985a9529af7d894db4815cd943546a58ebb847a Nov 02 17:44:21.723686 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:21.723664 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7"] Nov 02 17:44:21.800796 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:44:21.800769 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:44:22.371779 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:22.371300 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:44:22.371779 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:22.371386 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" podUID="64e985f1-f451-4ee1-ba69-76ce8949ca40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 17:44:22.529823 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:22.529789 4860 generic.go:356] "Generic (PLEG): container finished" podID="cd9777f9-d834-4e07-8493-6bd7f392be05" containerID="a8156fb909b3efbf0c813157814c2f36614fc35287910d0a8e599bfe041e0567" exitCode=0 Nov 02 17:44:22.530243 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:22.529994 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v" event={"ID":"cd9777f9-d834-4e07-8493-6bd7f392be05","Type":"ContainerDied","Data":"a8156fb909b3efbf0c813157814c2f36614fc35287910d0a8e599bfe041e0567"} Nov 02 17:44:22.531786 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:22.531758 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7" event={"ID":"80578a9c-ef49-49de-a95c-eb8d9e1a79bd","Type":"ContainerStarted","Data":"d27fe7c9af124f4dd4a0506947da14a1f09edbca0f95deed1b72cf6506732e20"} Nov 02 17:44:22.531938 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:22.531800 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7" event={"ID":"80578a9c-ef49-49de-a95c-eb8d9e1a79bd","Type":"ContainerStarted","Data":"add4b6cccc045c0674ef3e63f985a9529af7d894db4815cd943546a58ebb847a"} Nov 02 17:44:22.628660 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:22.628548 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7" podStartSLOduration=1.6285310800000001 podStartE2EDuration="1.62853108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:22.62736032 +0000 UTC m=+881.248720107" watchObservedRunningTime="2026-11-02 17:44:22.62853108 +0000 UTC m=+881.249890857" Nov 02 17:44:23.376874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.374279 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:44:23.376874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.374338 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" podUID="64e985f1-f451-4ee1-ba69-76ce8949ca40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 17:44:23.545813 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.545777 4860 generic.go:356] "Generic (PLEG): container finished" podID="80578a9c-ef49-49de-a95c-eb8d9e1a79bd" containerID="d27fe7c9af124f4dd4a0506947da14a1f09edbca0f95deed1b72cf6506732e20" exitCode=0 Nov 02 17:44:23.546276 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.545883 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7" event={"ID":"80578a9c-ef49-49de-a95c-eb8d9e1a79bd","Type":"ContainerDied","Data":"d27fe7c9af124f4dd4a0506947da14a1f09edbca0f95deed1b72cf6506732e20"} Nov 02 17:44:23.798019 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.796928 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v" Nov 02 17:44:23.852039 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.850451 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd9777f9-d834-4e07-8493-6bd7f392be05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9777f9-d834-4e07-8493-6bd7f392be05-maas-api-xks-ca\") pod \"cd9777f9-d834-4e07-8493-6bd7f392be05\" (UID: \"cd9777f9-d834-4e07-8493-6bd7f392be05\") " Nov 02 17:44:23.852039 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.850493 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd9777f9-d834-4e07-8493-6bd7f392be05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9777f9-d834-4e07-8493-6bd7f392be05-maas-api-openshift-ca\") pod \"cd9777f9-d834-4e07-8493-6bd7f392be05\" (UID: \"cd9777f9-d834-4e07-8493-6bd7f392be05\") " Nov 02 17:44:23.852039 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.850591 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9777f9-d834-4e07-8493-6bd7f392be05-kube-api-access-99zq8\" (UniqueName: \"kubernetes.io/projected/cd9777f9-d834-4e07-8493-6bd7f392be05-kube-api-access-99zq8\") pod \"cd9777f9-d834-4e07-8493-6bd7f392be05\" (UID: \"cd9777f9-d834-4e07-8493-6bd7f392be05\") " Nov 02 17:44:23.873815 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.873737 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9777f9-d834-4e07-8493-6bd7f392be05-kube-api-access-99zq8" pod "cd9777f9-d834-4e07-8493-6bd7f392be05" (UID: "cd9777f9-d834-4e07-8493-6bd7f392be05"). InnerVolumeSpecName "kube-api-access-99zq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:23.886960 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.886007 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd9777f9-d834-4e07-8493-6bd7f392be05-maas-api-xks-ca" pod "cd9777f9-d834-4e07-8493-6bd7f392be05" (UID: "cd9777f9-d834-4e07-8493-6bd7f392be05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:44:23.886960 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.886241 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9777f9-d834-4e07-8493-6bd7f392be05-maas-api-openshift-ca" pod "cd9777f9-d834-4e07-8493-6bd7f392be05" (UID: "cd9777f9-d834-4e07-8493-6bd7f392be05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:44:23.910907 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.910875 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g"] Nov 02 17:44:23.911679 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.911658 4860 cpu_manager.go:414] "containerMap: removing container" podUID="cd9777f9-d834-4e07-8493-6bd7f392be05" containerName="revoke-keys" Nov 02 17:44:23.911743 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.911682 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd9777f9-d834-4e07-8493-6bd7f392be05" containerName="revoke-keys" Nov 02 17:44:23.911834 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.911824 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd9777f9-d834-4e07-8493-6bd7f392be05" containerName="revoke-keys" Nov 02 17:44:23.915951 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.915924 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g" Nov 02 17:44:23.942124 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.942091 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g"] Nov 02 17:44:23.953202 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.952655 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf5nh\" (UniqueName: \"kubernetes.io/projected/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-kube-api-access-nf5nh\") pod \"maas-api-revoke-sub-093569f7-n2c8g\" (UID: \"b64d9245-f8e9-48b9-a762-7c6c187a7ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g" Nov 02 17:44:23.953202 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.952705 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-093569f7-n2c8g\" (UID: \"b64d9245-f8e9-48b9-a762-7c6c187a7ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g" Nov 02 17:44:23.953202 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.952781 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-093569f7-n2c8g\" (UID: \"b64d9245-f8e9-48b9-a762-7c6c187a7ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g" Nov 02 17:44:23.953202 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.953091 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9777f9-d834-4e07-8493-6bd7f392be05-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:23.953202 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.953109 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9777f9-d834-4e07-8493-6bd7f392be05-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:23.953202 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:23.953123 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99zq8\" (UniqueName: \"kubernetes.io/projected/cd9777f9-d834-4e07-8493-6bd7f392be05-kube-api-access-99zq8\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:24.085875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.064332 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf5nh\" (UniqueName: \"kubernetes.io/projected/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-kube-api-access-nf5nh\") pod \"maas-api-revoke-sub-093569f7-n2c8g\" (UID: \"b64d9245-f8e9-48b9-a762-7c6c187a7ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g" Nov 02 17:44:24.085875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.064399 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-093569f7-n2c8g\" (UID: \"b64d9245-f8e9-48b9-a762-7c6c187a7ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g" Nov 02 17:44:24.085875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.064502 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-093569f7-n2c8g\" (UID: \"b64d9245-f8e9-48b9-a762-7c6c187a7ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g" Nov 02 17:44:24.085875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.080885 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-093569f7-n2c8g\" (UID: \"b64d9245-f8e9-48b9-a762-7c6c187a7ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g" Nov 02 17:44:24.087728 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.087699 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-093569f7-n2c8g\" (UID: \"b64d9245-f8e9-48b9-a762-7c6c187a7ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g" Nov 02 17:44:24.104518 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.104269 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf5nh\" (UniqueName: \"kubernetes.io/projected/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-kube-api-access-nf5nh\") pod \"maas-api-revoke-sub-093569f7-n2c8g\" (UID: \"b64d9245-f8e9-48b9-a762-7c6c187a7ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g" Nov 02 17:44:24.244705 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.244625 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g" Nov 02 17:44:24.378893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.376234 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:44:24.378893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.376296 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" podUID="64e985f1-f451-4ee1-ba69-76ce8949ca40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 17:44:24.581983 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.581958 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v" Nov 02 17:44:24.582529 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.581997 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v" event={"ID":"cd9777f9-d834-4e07-8493-6bd7f392be05","Type":"ContainerDied","Data":"0b597a388bdc003f13a90ab162f3e3682759d9e5b5ace073b719594b58a743e8"} Nov 02 17:44:24.582529 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.582033 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b597a388bdc003f13a90ab162f3e3682759d9e5b5ace073b719594b58a743e8" Nov 02 17:44:24.588451 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.587719 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g"] Nov 02 17:44:24.590346 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:44:24.590320 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64d9245_f8e9_48b9_a762_7c6c187a7ae3.slice/crio-648564ad782b4a01a60e3573577ced8dd7c6db82081fa14d7793ff8572828332 WatchSource:0}: Error finding container 648564ad782b4a01a60e3573577ced8dd7c6db82081fa14d7793ff8572828332: Status 404 returned error can't find the container with id 648564ad782b4a01a60e3573577ced8dd7c6db82081fa14d7793ff8572828332 Nov 02 17:44:24.894877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.894595 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7" Nov 02 17:44:24.999350 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.994037 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-maas-api-xks-ca\") pod \"80578a9c-ef49-49de-a95c-eb8d9e1a79bd\" (UID: \"80578a9c-ef49-49de-a95c-eb8d9e1a79bd\") " Nov 02 17:44:24.999350 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.994178 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-kube-api-access-2x7gc\" (UniqueName: \"kubernetes.io/projected/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-kube-api-access-2x7gc\") pod \"80578a9c-ef49-49de-a95c-eb8d9e1a79bd\" (UID: \"80578a9c-ef49-49de-a95c-eb8d9e1a79bd\") " Nov 02 17:44:24.999350 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.994340 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-maas-api-openshift-ca\") pod \"80578a9c-ef49-49de-a95c-eb8d9e1a79bd\" (UID: \"80578a9c-ef49-49de-a95c-eb8d9e1a79bd\") " Nov 02 17:44:24.999350 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.995178 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-maas-api-openshift-ca" pod "80578a9c-ef49-49de-a95c-eb8d9e1a79bd" (UID: "80578a9c-ef49-49de-a95c-eb8d9e1a79bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:44:24.999350 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.995309 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:24.999350 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.998150 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-kube-api-access-2x7gc" pod "80578a9c-ef49-49de-a95c-eb8d9e1a79bd" (UID: "80578a9c-ef49-49de-a95c-eb8d9e1a79bd"). InnerVolumeSpecName "kube-api-access-2x7gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:24.999350 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:24.999040 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-maas-api-xks-ca" pod "80578a9c-ef49-49de-a95c-eb8d9e1a79bd" (UID: "80578a9c-ef49-49de-a95c-eb8d9e1a79bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:44:25.096659 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.096623 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:25.097127 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.096888 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x7gc\" (UniqueName: \"kubernetes.io/projected/80578a9c-ef49-49de-a95c-eb8d9e1a79bd-kube-api-access-2x7gc\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:25.210180 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.210155 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:44:25.299948 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.299912 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-token\" (UniqueName: \"kubernetes.io/projected/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-token\") pod \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " Nov 02 17:44:25.300109 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.300037 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-podinfo\") pod \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " Nov 02 17:44:25.300188 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.300170 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-data\") pod \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " Nov 02 17:44:25.300245 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.300213 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d6284d8-6036-4088-a7e0-a1d902f27ecb-kube-api-access-zr22w\" (UniqueName: \"kubernetes.io/projected/6d6284d8-6036-4088-a7e0-a1d902f27ecb-kube-api-access-zr22w\") pod \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " Nov 02 17:44:25.300530 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.300505 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-data" pod "6d6284d8-6036-4088-a7e0-a1d902f27ecb" (UID: "6d6284d8-6036-4088-a7e0-a1d902f27ecb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:44:25.300737 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.300716 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-workload-certs\") pod \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " Nov 02 17:44:25.300814 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.300757 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-envoy\") pod \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " Nov 02 17:44:25.300926 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.300909 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istiod-ca-cert\") pod \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " Nov 02 17:44:25.300997 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.300950 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-credential-socket\") pod \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " Nov 02 17:44:25.300997 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.300987 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-workload-socket\") pod \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\" (UID: \"6d6284d8-6036-4088-a7e0-a1d902f27ecb\") " Nov 02 17:44:25.302348 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.302308 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-workload-certs" pod "6d6284d8-6036-4088-a7e0-a1d902f27ecb" (UID: "6d6284d8-6036-4088-a7e0-a1d902f27ecb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:44:25.302927 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.302894 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-podinfo" pod "6d6284d8-6036-4088-a7e0-a1d902f27ecb" (UID: "6d6284d8-6036-4088-a7e0-a1d902f27ecb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:44:25.303103 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.303077 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-credential-socket" pod "6d6284d8-6036-4088-a7e0-a1d902f27ecb" (UID: "6d6284d8-6036-4088-a7e0-a1d902f27ecb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:44:25.303321 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.303296 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istiod-ca-cert" pod "6d6284d8-6036-4088-a7e0-a1d902f27ecb" (UID: "6d6284d8-6036-4088-a7e0-a1d902f27ecb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:44:25.303403 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.303304 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-workload-socket" pod "6d6284d8-6036-4088-a7e0-a1d902f27ecb" (UID: "6d6284d8-6036-4088-a7e0-a1d902f27ecb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:44:25.303871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.303818 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-token" pod "6d6284d8-6036-4088-a7e0-a1d902f27ecb" (UID: "6d6284d8-6036-4088-a7e0-a1d902f27ecb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:25.316370 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.316340 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:25.316488 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.316374 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:25.316488 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.316388 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:25.316488 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.316407 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:25.316488 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.316422 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:25.316488 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.316435 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:25.316488 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.316449 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:25.318560 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.318522 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6284d8-6036-4088-a7e0-a1d902f27ecb-kube-api-access-zr22w" pod "6d6284d8-6036-4088-a7e0-a1d902f27ecb" (UID: "6d6284d8-6036-4088-a7e0-a1d902f27ecb"). InnerVolumeSpecName "kube-api-access-zr22w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:25.320577 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.320546 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-envoy" pod "6d6284d8-6036-4088-a7e0-a1d902f27ecb" (UID: "6d6284d8-6036-4088-a7e0-a1d902f27ecb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:44:25.371873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.371355 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:44:25.371873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.371422 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" podUID="64e985f1-f451-4ee1-ba69-76ce8949ca40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 17:44:25.418679 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.418586 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr22w\" (UniqueName: \"kubernetes.io/projected/6d6284d8-6036-4088-a7e0-a1d902f27ecb-kube-api-access-zr22w\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:25.418679 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.418622 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6284d8-6036-4088-a7e0-a1d902f27ecb-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:25.596437 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.596397 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7" event={"ID":"80578a9c-ef49-49de-a95c-eb8d9e1a79bd","Type":"ContainerDied","Data":"add4b6cccc045c0674ef3e63f985a9529af7d894db4815cd943546a58ebb847a"} Nov 02 17:44:25.596437 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.596420 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7" Nov 02 17:44:25.596957 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.596462 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="add4b6cccc045c0674ef3e63f985a9529af7d894db4815cd943546a58ebb847a" Nov 02 17:44:25.598410 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.598384 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g" event={"ID":"b64d9245-f8e9-48b9-a762-7c6c187a7ae3","Type":"ContainerStarted","Data":"0c232439398b5394e2e5c093aa26dea009123a0cab7dde5c3459aefce6b5b3a8"} Nov 02 17:44:25.598543 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.598430 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g" event={"ID":"b64d9245-f8e9-48b9-a762-7c6c187a7ae3","Type":"ContainerStarted","Data":"648564ad782b4a01a60e3573577ced8dd7c6db82081fa14d7793ff8572828332"} Nov 02 17:44:25.600963 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.600945 4860 generic.go:356] "Generic (PLEG): container finished" podID="6d6284d8-6036-4088-a7e0-a1d902f27ecb" containerID="589d2ccd71935caf23ff4b0c79740268505c3699ba457eca5c3fa94cb67c335c" exitCode=0 Nov 02 17:44:25.601108 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.601001 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" Nov 02 17:44:25.601262 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.601007 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" event={"ID":"6d6284d8-6036-4088-a7e0-a1d902f27ecb","Type":"ContainerDied","Data":"589d2ccd71935caf23ff4b0c79740268505c3699ba457eca5c3fa94cb67c335c"} Nov 02 17:44:25.601316 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.601287 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb" event={"ID":"6d6284d8-6036-4088-a7e0-a1d902f27ecb","Type":"ContainerDied","Data":"114bb94bc3ec337561879b83ef25841f1de516c9bfc04d9da613e63dbabbe44c"} Nov 02 17:44:25.601316 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.601310 4860 scope.go:122] "RemoveContainer" containerID="589d2ccd71935caf23ff4b0c79740268505c3699ba457eca5c3fa94cb67c335c" Nov 02 17:44:25.612759 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.612744 4860 scope.go:122] "RemoveContainer" containerID="589d2ccd71935caf23ff4b0c79740268505c3699ba457eca5c3fa94cb67c335c" Nov 02 17:44:25.613050 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:44:25.613030 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"589d2ccd71935caf23ff4b0c79740268505c3699ba457eca5c3fa94cb67c335c\": container with ID starting with 589d2ccd71935caf23ff4b0c79740268505c3699ba457eca5c3fa94cb67c335c not found: ID does not exist" containerID="589d2ccd71935caf23ff4b0c79740268505c3699ba457eca5c3fa94cb67c335c" Nov 02 17:44:25.613117 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.613058 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"589d2ccd71935caf23ff4b0c79740268505c3699ba457eca5c3fa94cb67c335c"} err="failed to get container status \"589d2ccd71935caf23ff4b0c79740268505c3699ba457eca5c3fa94cb67c335c\": rpc error: code = NotFound desc = could not find container \"589d2ccd71935caf23ff4b0c79740268505c3699ba457eca5c3fa94cb67c335c\": container with ID starting with 589d2ccd71935caf23ff4b0c79740268505c3699ba457eca5c3fa94cb67c335c not found: ID does not exist" Nov 02 17:44:25.649476 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.649409 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g" podStartSLOduration=2.649389003 podStartE2EDuration="2.649389003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:25.642046011 +0000 UTC m=+884.263405788" watchObservedRunningTime="2026-11-02 17:44:25.649389003 +0000 UTC m=+884.270748781" Nov 02 17:44:25.710643 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.710612 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb"] Nov 02 17:44:25.716183 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:25.716157 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qlptb"] Nov 02 17:44:26.379814 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:26.373361 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:44:26.379814 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:26.373422 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" podUID="64e985f1-f451-4ee1-ba69-76ce8949ca40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 17:44:26.612006 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:26.611960 4860 generic.go:356] "Generic (PLEG): container finished" podID="b64d9245-f8e9-48b9-a762-7c6c187a7ae3" containerID="0c232439398b5394e2e5c093aa26dea009123a0cab7dde5c3459aefce6b5b3a8" exitCode=0 Nov 02 17:44:26.612425 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:26.612091 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g" event={"ID":"b64d9245-f8e9-48b9-a762-7c6c187a7ae3","Type":"ContainerDied","Data":"0c232439398b5394e2e5c093aa26dea009123a0cab7dde5c3459aefce6b5b3a8"} Nov 02 17:44:27.370522 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:27.370478 4860 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:44:27.370706 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:27.370537 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" podUID="64e985f1-f451-4ee1-ba69-76ce8949ca40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 17:44:27.504494 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:27.504452 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d6284d8-6036-4088-a7e0-a1d902f27ecb" path="/var/lib/kubelet/pods/6d6284d8-6036-4088-a7e0-a1d902f27ecb/volumes" Nov 02 17:44:27.826180 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:27.826151 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g" Nov 02 17:44:27.962104 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:27.962012 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-maas-api-xks-ca\") pod \"b64d9245-f8e9-48b9-a762-7c6c187a7ae3\" (UID: \"b64d9245-f8e9-48b9-a762-7c6c187a7ae3\") " Nov 02 17:44:27.962260 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:27.962168 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-kube-api-access-nf5nh\" (UniqueName: \"kubernetes.io/projected/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-kube-api-access-nf5nh\") pod \"b64d9245-f8e9-48b9-a762-7c6c187a7ae3\" (UID: \"b64d9245-f8e9-48b9-a762-7c6c187a7ae3\") " Nov 02 17:44:27.962260 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:27.962200 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-maas-api-openshift-ca\") pod \"b64d9245-f8e9-48b9-a762-7c6c187a7ae3\" (UID: \"b64d9245-f8e9-48b9-a762-7c6c187a7ae3\") " Nov 02 17:44:27.962711 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:27.962681 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-maas-api-openshift-ca" pod "b64d9245-f8e9-48b9-a762-7c6c187a7ae3" (UID: "b64d9245-f8e9-48b9-a762-7c6c187a7ae3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:44:27.962833 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:27.962786 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:27.964206 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:27.964180 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-maas-api-xks-ca" pod "b64d9245-f8e9-48b9-a762-7c6c187a7ae3" (UID: "b64d9245-f8e9-48b9-a762-7c6c187a7ae3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:44:27.964647 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:27.964629 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-kube-api-access-nf5nh" pod "b64d9245-f8e9-48b9-a762-7c6c187a7ae3" (UID: "b64d9245-f8e9-48b9-a762-7c6c187a7ae3"). InnerVolumeSpecName "kube-api-access-nf5nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:28.064769 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:28.064732 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf5nh\" (UniqueName: \"kubernetes.io/projected/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-kube-api-access-nf5nh\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:28.064769 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:28.064767 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64d9245-f8e9-48b9-a762-7c6c187a7ae3-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:28.374409 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:28.374377 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:28.374906 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:28.374881 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:28.375680 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:28.375663 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:44:28.641169 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:28.641074 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g" Nov 02 17:44:28.641341 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:28.641079 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g" event={"ID":"b64d9245-f8e9-48b9-a762-7c6c187a7ae3","Type":"ContainerDied","Data":"648564ad782b4a01a60e3573577ced8dd7c6db82081fa14d7793ff8572828332"} Nov 02 17:44:28.641341 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:28.641217 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="648564ad782b4a01a60e3573577ced8dd7c6db82081fa14d7793ff8572828332" Nov 02 17:44:30.534996 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.534947 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp"] Nov 02 17:44:30.535652 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.535637 4860 cpu_manager.go:414] "containerMap: removing container" podUID="80578a9c-ef49-49de-a95c-eb8d9e1a79bd" containerName="revoke-keys" Nov 02 17:44:30.535709 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.535653 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80578a9c-ef49-49de-a95c-eb8d9e1a79bd" containerName="revoke-keys" Nov 02 17:44:30.535709 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.535697 4860 cpu_manager.go:414] "containerMap: removing container" podUID="6d6284d8-6036-4088-a7e0-a1d902f27ecb" containerName="istio-proxy" Nov 02 17:44:30.535709 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.535703 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d6284d8-6036-4088-a7e0-a1d902f27ecb" containerName="istio-proxy" Nov 02 17:44:30.535809 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.535714 4860 cpu_manager.go:414] "containerMap: removing container" podUID="b64d9245-f8e9-48b9-a762-7c6c187a7ae3" containerName="revoke-keys" Nov 02 17:44:30.535809 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.535719 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64d9245-f8e9-48b9-a762-7c6c187a7ae3" containerName="revoke-keys" Nov 02 17:44:30.535877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.535842 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="80578a9c-ef49-49de-a95c-eb8d9e1a79bd" containerName="revoke-keys" Nov 02 17:44:30.535877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.535870 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d6284d8-6036-4088-a7e0-a1d902f27ecb" containerName="istio-proxy" Nov 02 17:44:30.535938 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.535878 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64d9245-f8e9-48b9-a762-7c6c187a7ae3" containerName="revoke-keys" Nov 02 17:44:30.539749 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.539727 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp" Nov 02 17:44:30.567896 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.567863 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp"] Nov 02 17:44:30.573549 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.573519 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:44:30.573676 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.573631 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:44:30.598919 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.598828 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894756e3-8380-48c0-8881-0bfa433893f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13938481-tlknp\" (UID: \"894756e3-8380-48c0-8881-0bfa433893f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp" Nov 02 17:44:30.599117 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.598984 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmgvb\" (UniqueName: \"kubernetes.io/projected/894756e3-8380-48c0-8881-0bfa433893f0-kube-api-access-zmgvb\") pod \"maas-api-revoke-sub-13938481-tlknp\" (UID: \"894756e3-8380-48c0-8881-0bfa433893f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp" Nov 02 17:44:30.599212 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.599185 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894756e3-8380-48c0-8881-0bfa433893f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13938481-tlknp\" (UID: \"894756e3-8380-48c0-8881-0bfa433893f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp" Nov 02 17:44:30.700260 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.700228 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894756e3-8380-48c0-8881-0bfa433893f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13938481-tlknp\" (UID: \"894756e3-8380-48c0-8881-0bfa433893f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp" Nov 02 17:44:30.700439 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.700333 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894756e3-8380-48c0-8881-0bfa433893f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13938481-tlknp\" (UID: \"894756e3-8380-48c0-8881-0bfa433893f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp" Nov 02 17:44:30.700439 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.700423 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmgvb\" (UniqueName: \"kubernetes.io/projected/894756e3-8380-48c0-8881-0bfa433893f0-kube-api-access-zmgvb\") pod \"maas-api-revoke-sub-13938481-tlknp\" (UID: \"894756e3-8380-48c0-8881-0bfa433893f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp" Nov 02 17:44:30.701166 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.701137 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894756e3-8380-48c0-8881-0bfa433893f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13938481-tlknp\" (UID: \"894756e3-8380-48c0-8881-0bfa433893f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp" Nov 02 17:44:30.703143 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.703117 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894756e3-8380-48c0-8881-0bfa433893f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13938481-tlknp\" (UID: \"894756e3-8380-48c0-8881-0bfa433893f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp" Nov 02 17:44:30.728742 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.728710 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmgvb\" (UniqueName: \"kubernetes.io/projected/894756e3-8380-48c0-8881-0bfa433893f0-kube-api-access-zmgvb\") pod \"maas-api-revoke-sub-13938481-tlknp\" (UID: \"894756e3-8380-48c0-8881-0bfa433893f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp" Nov 02 17:44:30.854622 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:30.854592 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp" Nov 02 17:44:31.170930 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:44:31.170899 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894756e3_8380_48c0_8881_0bfa433893f0.slice/crio-6b7ba27b8336ba426338aa594a5d8ab54c29e16c5741ca0e023111188d3a8deb WatchSource:0}: Error finding container 6b7ba27b8336ba426338aa594a5d8ab54c29e16c5741ca0e023111188d3a8deb: Status 404 returned error can't find the container with id 6b7ba27b8336ba426338aa594a5d8ab54c29e16c5741ca0e023111188d3a8deb Nov 02 17:44:31.173286 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:31.173259 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp"] Nov 02 17:44:31.669028 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:31.668979 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp" event={"ID":"894756e3-8380-48c0-8881-0bfa433893f0","Type":"ContainerStarted","Data":"2c7b548f1b2202ec1a2407168d404f489a1d27f09f52e29011a04f91f1fae80c"} Nov 02 17:44:31.669389 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:31.669043 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp" event={"ID":"894756e3-8380-48c0-8881-0bfa433893f0","Type":"ContainerStarted","Data":"6b7ba27b8336ba426338aa594a5d8ab54c29e16c5741ca0e023111188d3a8deb"} Nov 02 17:44:31.717665 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:31.717564 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp" podStartSLOduration=1.7175484760000002 podStartE2EDuration="1.717548476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:31.711685137 +0000 UTC m=+890.333044925" watchObservedRunningTime="2026-11-02 17:44:31.717548476 +0000 UTC m=+890.338908251" Nov 02 17:44:32.679463 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:32.679428 4860 generic.go:356] "Generic (PLEG): container finished" podID="894756e3-8380-48c0-8881-0bfa433893f0" containerID="2c7b548f1b2202ec1a2407168d404f489a1d27f09f52e29011a04f91f1fae80c" exitCode=0 Nov 02 17:44:32.679900 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:32.679521 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp" event={"ID":"894756e3-8380-48c0-8881-0bfa433893f0","Type":"ContainerDied","Data":"2c7b548f1b2202ec1a2407168d404f489a1d27f09f52e29011a04f91f1fae80c"} Nov 02 17:44:33.853926 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:33.853893 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp" Nov 02 17:44:33.950346 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:33.950249 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/894756e3-8380-48c0-8881-0bfa433893f0-kube-api-access-zmgvb\" (UniqueName: \"kubernetes.io/projected/894756e3-8380-48c0-8881-0bfa433893f0-kube-api-access-zmgvb\") pod \"894756e3-8380-48c0-8881-0bfa433893f0\" (UID: \"894756e3-8380-48c0-8881-0bfa433893f0\") " Nov 02 17:44:33.950531 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:33.950368 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/894756e3-8380-48c0-8881-0bfa433893f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894756e3-8380-48c0-8881-0bfa433893f0-maas-api-openshift-ca\") pod \"894756e3-8380-48c0-8881-0bfa433893f0\" (UID: \"894756e3-8380-48c0-8881-0bfa433893f0\") " Nov 02 17:44:33.950531 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:33.950477 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/894756e3-8380-48c0-8881-0bfa433893f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894756e3-8380-48c0-8881-0bfa433893f0-maas-api-xks-ca\") pod \"894756e3-8380-48c0-8881-0bfa433893f0\" (UID: \"894756e3-8380-48c0-8881-0bfa433893f0\") " Nov 02 17:44:33.951315 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:33.951288 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/894756e3-8380-48c0-8881-0bfa433893f0-maas-api-openshift-ca" pod "894756e3-8380-48c0-8881-0bfa433893f0" (UID: "894756e3-8380-48c0-8881-0bfa433893f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:44:33.953278 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:33.953247 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/894756e3-8380-48c0-8881-0bfa433893f0-maas-api-xks-ca" pod "894756e3-8380-48c0-8881-0bfa433893f0" (UID: "894756e3-8380-48c0-8881-0bfa433893f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:44:33.953396 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:33.953276 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894756e3-8380-48c0-8881-0bfa433893f0-kube-api-access-zmgvb" pod "894756e3-8380-48c0-8881-0bfa433893f0" (UID: "894756e3-8380-48c0-8881-0bfa433893f0"). InnerVolumeSpecName "kube-api-access-zmgvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:34.053020 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:34.052985 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmgvb\" (UniqueName: \"kubernetes.io/projected/894756e3-8380-48c0-8881-0bfa433893f0-kube-api-access-zmgvb\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:34.053020 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:34.053016 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894756e3-8380-48c0-8881-0bfa433893f0-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:34.053020 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:34.053028 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894756e3-8380-48c0-8881-0bfa433893f0-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:34.698253 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:34.698220 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp" Nov 02 17:44:34.698398 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:34.698220 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp" event={"ID":"894756e3-8380-48c0-8881-0bfa433893f0","Type":"ContainerDied","Data":"6b7ba27b8336ba426338aa594a5d8ab54c29e16c5741ca0e023111188d3a8deb"} Nov 02 17:44:34.698398 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:34.698339 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b7ba27b8336ba426338aa594a5d8ab54c29e16c5741ca0e023111188d3a8deb" Nov 02 17:44:41.645801 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:41.645771 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 17:44:41.650370 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:41.650343 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 17:44:41.664392 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:41.664362 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 17:44:41.674256 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:41.674226 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 17:44:44.833907 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:44.833874 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x"] Nov 02 17:44:44.834618 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:44.834602 4860 cpu_manager.go:414] "containerMap: removing container" podUID="894756e3-8380-48c0-8881-0bfa433893f0" containerName="revoke-keys" Nov 02 17:44:44.834669 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:44.834620 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="894756e3-8380-48c0-8881-0bfa433893f0" containerName="revoke-keys" Nov 02 17:44:44.834764 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:44.834755 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="894756e3-8380-48c0-8881-0bfa433893f0" containerName="revoke-keys" Nov 02 17:44:44.840197 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:44.840175 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x" Nov 02 17:44:44.871879 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:44.859333 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x"] Nov 02 17:44:44.871879 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:44.864268 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:44:44.871879 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:44.864286 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:44:45.012863 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:45.012826 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg44z\" (UniqueName: \"kubernetes.io/projected/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-kube-api-access-tg44z\") pod \"maas-api-revoke-sub-b5c20415-bwb8x\" (UID: \"0ebc34e1-d76e-42e6-8f14-a29d41a93f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x" Nov 02 17:44:45.013042 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:45.012926 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5c20415-bwb8x\" (UID: \"0ebc34e1-d76e-42e6-8f14-a29d41a93f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x" Nov 02 17:44:45.013042 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:45.013001 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5c20415-bwb8x\" (UID: \"0ebc34e1-d76e-42e6-8f14-a29d41a93f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x" Nov 02 17:44:45.157917 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:45.153349 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg44z\" (UniqueName: \"kubernetes.io/projected/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-kube-api-access-tg44z\") pod \"maas-api-revoke-sub-b5c20415-bwb8x\" (UID: \"0ebc34e1-d76e-42e6-8f14-a29d41a93f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x" Nov 02 17:44:45.157917 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:45.153502 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5c20415-bwb8x\" (UID: \"0ebc34e1-d76e-42e6-8f14-a29d41a93f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x" Nov 02 17:44:45.157917 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:45.153616 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5c20415-bwb8x\" (UID: \"0ebc34e1-d76e-42e6-8f14-a29d41a93f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x" Nov 02 17:44:45.157917 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:45.155419 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5c20415-bwb8x\" (UID: \"0ebc34e1-d76e-42e6-8f14-a29d41a93f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x" Nov 02 17:44:45.182218 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:45.182141 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5c20415-bwb8x\" (UID: \"0ebc34e1-d76e-42e6-8f14-a29d41a93f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x" Nov 02 17:44:45.231462 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:45.216595 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg44z\" (UniqueName: \"kubernetes.io/projected/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-kube-api-access-tg44z\") pod \"maas-api-revoke-sub-b5c20415-bwb8x\" (UID: \"0ebc34e1-d76e-42e6-8f14-a29d41a93f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x" Nov 02 17:44:45.460956 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:45.456114 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x" Nov 02 17:44:46.643928 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:46.626639 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:46.779257 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:46.773569 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:46.779257 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:46.776130 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 02 17:44:46.877890 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:46.840988 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns"] Nov 02 17:44:46.877890 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:46.857940 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:46.922136 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:46.907375 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.014875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:46.927317 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.014875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:46.976344 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.014875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:46.976425 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 02 17:44:47.014875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:46.977581 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.016354 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.016311 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.017560 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.017532 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 02 17:44:47.089880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.085583 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db49741-b199-403e-98f4-fbba98d95b84-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.089880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.085719 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.106934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.090166 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.106934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.090244 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db49741-b199-403e-98f4-fbba98d95b84-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.106934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.090275 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95v29\" (UniqueName: \"kubernetes.io/projected/4db49741-b199-403e-98f4-fbba98d95b84-kube-api-access-95v29\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.106934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.090323 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.106934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.090371 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.106934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.090400 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db49741-b199-403e-98f4-fbba98d95b84-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.106934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.090494 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.156362 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.152780 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.156362 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.153113 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.156362 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.153975 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.172988 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.172897 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns"] Nov 02 17:44:47.218880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.198890 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db49741-b199-403e-98f4-fbba98d95b84-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.218880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.198990 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.218880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.199038 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db49741-b199-403e-98f4-fbba98d95b84-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.218880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.199148 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.218880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.199282 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.218880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.199315 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db49741-b199-403e-98f4-fbba98d95b84-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.218880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.199343 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95v29\" (UniqueName: \"kubernetes.io/projected/4db49741-b199-403e-98f4-fbba98d95b84-kube-api-access-95v29\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.218880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.199385 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.218880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.199427 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.218880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.202206 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.218880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.202687 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.218880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.204352 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.218880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.204948 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.218880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.206187 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db49741-b199-403e-98f4-fbba98d95b84-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.227499 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.227435 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.292459 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.274569 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95v29\" (UniqueName: \"kubernetes.io/projected/4db49741-b199-403e-98f4-fbba98d95b84-kube-api-access-95v29\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.316883 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.313591 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db49741-b199-403e-98f4-fbba98d95b84-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.429382 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.363016 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.429382 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.394740 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db49741-b199-403e-98f4-fbba98d95b84-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.429382 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.407018 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.429382 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.418945 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd"] Nov 02 17:44:47.429382 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.428389 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.502150 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.502111 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:47.580964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.549706 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.580964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.549826 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.580964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.549881 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49146668-6dbb-4df5-a1c1-b2c150a74979-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.580964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.549955 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwj74\" (UniqueName: \"kubernetes.io/projected/49146668-6dbb-4df5-a1c1-b2c150a74979-kube-api-access-gwj74\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.580964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.550033 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.580964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.550060 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.580964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.550105 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.580964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.550154 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.580964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.550369 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.637280 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.629919 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.704697 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.704488 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd"] Nov 02 17:44:47.704697 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.704609 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:44:47.749633 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.749601 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.749929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.749906 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.750084 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.750066 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49146668-6dbb-4df5-a1c1-b2c150a74979-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.750256 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.750240 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwj74\" (UniqueName: \"kubernetes.io/projected/49146668-6dbb-4df5-a1c1-b2c150a74979-kube-api-access-gwj74\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.750403 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.750387 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.750524 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.750508 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.750670 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.750655 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.750791 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.750777 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.751168 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.751144 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.763880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.752811 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.764097 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.754788 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.764213 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.755085 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.764315 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.761466 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.774985 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.774931 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.780299 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.780276 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:44:47.792872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.780979 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.794457 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.794433 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:44:47.819477 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.785114 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49146668-6dbb-4df5-a1c1-b2c150a74979-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.904765 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.904719 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwj74\" (UniqueName: \"kubernetes.io/projected/49146668-6dbb-4df5-a1c1-b2c150a74979-kube-api-access-gwj74\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.927880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.920456 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:47.933146 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.933105 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.938469 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.938442 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:44:47.994160 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.977345 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 02 17:44:47.994160 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.977778 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:44:47.994160 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.978803 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:47.994160 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.979137 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:44:47.997867 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.994632 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:48.006885 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:47.998400 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:44:48.033691 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.020062 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:48.034323 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.034294 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:44:48.088879 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.087526 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:48.130041 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.130007 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x"] Nov 02 17:44:48.206874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.168967 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:48.206874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.169331 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:48.206874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.169597 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 02 17:44:48.206874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.169885 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:44:48.206874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.169942 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:44:48.206874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.169993 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:44:48.206874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.181647 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:48.372420 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.372388 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:44:48.460793 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.430015 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:48.576069 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.572979 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Nov 02 17:44:48.576069 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.573382 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 02 17:44:48.688902 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.601059 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:44:48.747775 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.747074 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:44:48.747775 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.679495 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 02 17:44:48.816954 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.784974 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:48.850939 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:48.843925 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 02 17:44:49.160441 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:49.037419 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 02 17:44:49.160441 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:49.039185 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:49.160441 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:49.156005 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 02 17:44:49.211875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:49.171795 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:49.211875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:49.172182 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 02 17:44:49.211875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:49.172454 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 17:44:49.211875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:49.209364 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:49.674655 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:49.582968 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:49.674655 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:49.604061 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:49.698299 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:49.604020 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:49.959288 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:49.959174 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Nov 02 17:44:50.603755 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.595506 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:50.603755 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.032209 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:50.603755 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.599958 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:50.603755 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.048238 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 02 17:44:50.603755 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.228001 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 02 17:44:50.603755 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.112247 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:50.604919 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.228071 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:50.621034 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.062571 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:50.622000 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.572765 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" podUID="840ac652-3ceb-43ef-91b3-42c19179fe01" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 02 17:44:50.622000 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.604976 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:50.622931 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.235930 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 17:44:50.672929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.672891 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 02 17:44:50.732271 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.447529 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 17:44:50.803720 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.799451 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:50.803720 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.799553 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:50.803720 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.799601 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:50.803720 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:44:50.801425 4860 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.155s" Nov 02 17:44:50.803720 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.801458 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x" event={"ID":"0ebc34e1-d76e-42e6-8f14-a29d41a93f5c","Type":"ContainerStarted","Data":"8502f199ba2712dad68d990553b26e808cd34e34680a80544f8f985c62a25fbe"} Nov 02 17:44:50.803720 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.801566 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:44:50.803720 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.801688 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:44:50.803720 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.801825 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:44:50.803720 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.801964 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:44:50.803720 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.803577 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:44:50.804280 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.803766 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:44:50.804280 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.803891 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:44:50.804280 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.803925 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:44:50.804280 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.804149 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:44:50.804464 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.804308 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:44:50.809899 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.806264 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:44:50.839155 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.829097 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:44:50.839155 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.837654 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:44:50.839365 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.839202 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:44:50.860573 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.860543 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" event={"ID":"4db49741-b199-403e-98f4-fbba98d95b84","Type":"ContainerStarted","Data":"e2243171f7b5a5b5d21cc24b74215957b96a208f54c6c1db0684f32d1693ac42"} Nov 02 17:44:50.873163 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.873134 4860 generic.go:356] "Generic (PLEG): container finished" podID="0ebc34e1-d76e-42e6-8f14-a29d41a93f5c" containerID="c2b085a793fa7983e85ce9cfe1bb65091bb7602809ce1365545471593cc788c3" exitCode=0 Nov 02 17:44:50.873682 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.873659 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x" event={"ID":"0ebc34e1-d76e-42e6-8f14-a29d41a93f5c","Type":"ContainerDied","Data":"c2b085a793fa7983e85ce9cfe1bb65091bb7602809ce1365545471593cc788c3"} Nov 02 17:44:50.914901 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.914837 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs"] Nov 02 17:44:50.921267 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.921238 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs" Nov 02 17:44:50.923025 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.923003 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs"] Nov 02 17:44:50.929567 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.929545 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns"] Nov 02 17:44:51.000988 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:50.999978 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd"] Nov 02 17:44:51.001901 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:51.001842 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:51.002021 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:51.001947 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:51.002393 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:51.002363 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:44:51.102788 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:51.102748 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e25244-8eaa-42e5-8b9f-3324fee7a288-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a49143b-j87xs\" (UID: \"20e25244-8eaa-42e5-8b9f-3324fee7a288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs" Nov 02 17:44:51.102982 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:51.102819 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e25244-8eaa-42e5-8b9f-3324fee7a288-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a49143b-j87xs\" (UID: \"20e25244-8eaa-42e5-8b9f-3324fee7a288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs" Nov 02 17:44:51.103048 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:51.103017 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hlf7\" (UniqueName: \"kubernetes.io/projected/20e25244-8eaa-42e5-8b9f-3324fee7a288-kube-api-access-9hlf7\") pod \"maas-api-revoke-sub-6a49143b-j87xs\" (UID: \"20e25244-8eaa-42e5-8b9f-3324fee7a288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs" Nov 02 17:44:51.213083 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:51.210786 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e25244-8eaa-42e5-8b9f-3324fee7a288-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a49143b-j87xs\" (UID: \"20e25244-8eaa-42e5-8b9f-3324fee7a288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs" Nov 02 17:44:51.213083 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:51.210886 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e25244-8eaa-42e5-8b9f-3324fee7a288-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a49143b-j87xs\" (UID: \"20e25244-8eaa-42e5-8b9f-3324fee7a288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs" Nov 02 17:44:51.213083 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:51.211112 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hlf7\" (UniqueName: \"kubernetes.io/projected/20e25244-8eaa-42e5-8b9f-3324fee7a288-kube-api-access-9hlf7\") pod \"maas-api-revoke-sub-6a49143b-j87xs\" (UID: \"20e25244-8eaa-42e5-8b9f-3324fee7a288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs" Nov 02 17:44:51.213083 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:51.212316 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e25244-8eaa-42e5-8b9f-3324fee7a288-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a49143b-j87xs\" (UID: \"20e25244-8eaa-42e5-8b9f-3324fee7a288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs" Nov 02 17:44:51.221904 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:51.221471 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e25244-8eaa-42e5-8b9f-3324fee7a288-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a49143b-j87xs\" (UID: \"20e25244-8eaa-42e5-8b9f-3324fee7a288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs" Nov 02 17:44:51.304437 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:51.304397 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hlf7\" (UniqueName: \"kubernetes.io/projected/20e25244-8eaa-42e5-8b9f-3324fee7a288-kube-api-access-9hlf7\") pod \"maas-api-revoke-sub-6a49143b-j87xs\" (UID: \"20e25244-8eaa-42e5-8b9f-3324fee7a288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs" Nov 02 17:44:51.552734 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:51.552654 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs" Nov 02 17:44:51.797649 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:44:51.797055 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:44:51.916176 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:51.909554 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs"] Nov 02 17:44:51.919842 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:51.919155 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" event={"ID":"4db49741-b199-403e-98f4-fbba98d95b84","Type":"ContainerStarted","Data":"9e8497a541e5cf41a39fcbd5046b00f03ae31d270e373f585bb8a04db3da9df5"} Nov 02 17:44:51.927964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:51.927930 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" event={"ID":"49146668-6dbb-4df5-a1c1-b2c150a74979","Type":"ContainerStarted","Data":"99f2acd3cb7d3cdb010c6a333b6fa60263c0b73440eb502805ea70e952e02461"} Nov 02 17:44:51.928067 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:51.927989 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" event={"ID":"49146668-6dbb-4df5-a1c1-b2c150a74979","Type":"ContainerStarted","Data":"1258e6fa1f38e52fd78e62da85e710b6ffaebe551437e6ffcc027f1596d14a04"} Nov 02 17:44:52.009750 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.008400 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" podStartSLOduration=6.008380313 podStartE2EDuration="6.008380313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:52.002469908 +0000 UTC m=+910.623829689" watchObservedRunningTime="2026-11-02 17:44:52.008380313 +0000 UTC m=+910.629740089" Nov 02 17:44:52.079909 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.077477 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" podStartSLOduration=6.077453033 podStartE2EDuration="6.077453033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:52.061933461 +0000 UTC m=+910.683293238" watchObservedRunningTime="2026-11-02 17:44:52.077453033 +0000 UTC m=+910.698812811" Nov 02 17:44:52.287110 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.284367 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x" Nov 02 17:44:52.343790 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.343758 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-maas-api-openshift-ca\") pod \"0ebc34e1-d76e-42e6-8f14-a29d41a93f5c\" (UID: \"0ebc34e1-d76e-42e6-8f14-a29d41a93f5c\") " Nov 02 17:44:52.343967 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.343893 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-kube-api-access-tg44z\" (UniqueName: \"kubernetes.io/projected/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-kube-api-access-tg44z\") pod \"0ebc34e1-d76e-42e6-8f14-a29d41a93f5c\" (UID: \"0ebc34e1-d76e-42e6-8f14-a29d41a93f5c\") " Nov 02 17:44:52.344407 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.344062 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-maas-api-xks-ca\") pod \"0ebc34e1-d76e-42e6-8f14-a29d41a93f5c\" (UID: \"0ebc34e1-d76e-42e6-8f14-a29d41a93f5c\") " Nov 02 17:44:52.344407 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.344294 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-maas-api-openshift-ca" pod "0ebc34e1-d76e-42e6-8f14-a29d41a93f5c" (UID: "0ebc34e1-d76e-42e6-8f14-a29d41a93f5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:44:52.345242 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.344844 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:52.347506 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.346842 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-kube-api-access-tg44z" pod "0ebc34e1-d76e-42e6-8f14-a29d41a93f5c" (UID: "0ebc34e1-d76e-42e6-8f14-a29d41a93f5c"). InnerVolumeSpecName "kube-api-access-tg44z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:52.347506 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.347066 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-maas-api-xks-ca" pod "0ebc34e1-d76e-42e6-8f14-a29d41a93f5c" (UID: "0ebc34e1-d76e-42e6-8f14-a29d41a93f5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:44:52.447002 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.446969 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg44z\" (UniqueName: \"kubernetes.io/projected/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-kube-api-access-tg44z\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:52.447002 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.447005 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:52.517086 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.515948 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:52.517086 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.516075 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:44:52.517086 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.516137 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" podUID="4db49741-b199-403e-98f4-fbba98d95b84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 02 17:44:52.582880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.581149 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:52.613042 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.613006 4860 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 02 17:44:52.613319 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.613272 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" podUID="49146668-6dbb-4df5-a1c1-b2c150a74979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 02 17:44:52.953489 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.953449 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x" event={"ID":"0ebc34e1-d76e-42e6-8f14-a29d41a93f5c","Type":"ContainerDied","Data":"8502f199ba2712dad68d990553b26e808cd34e34680a80544f8f985c62a25fbe"} Nov 02 17:44:52.953656 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.953575 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x" Nov 02 17:44:52.957874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.953971 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8502f199ba2712dad68d990553b26e808cd34e34680a80544f8f985c62a25fbe" Nov 02 17:44:53.004243 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.985175 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs" event={"ID":"20e25244-8eaa-42e5-8b9f-3324fee7a288","Type":"ContainerStarted","Data":"4009e24fe1459c92af99d12e102e62d2f088033bd3d3e5c0a6c22db77a12d42f"} Nov 02 17:44:53.004243 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:52.985235 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs" event={"ID":"20e25244-8eaa-42e5-8b9f-3324fee7a288","Type":"ContainerStarted","Data":"31d848efcda71c34c23ba3419fc4b55fd7a3d486dfd342b1371479969fd207f0"} Nov 02 17:44:53.049414 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:53.046491 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs" podStartSLOduration=4.046475128 podStartE2EDuration="4.046475128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:53.037490509 +0000 UTC m=+911.658850287" watchObservedRunningTime="2026-11-02 17:44:53.046475128 +0000 UTC m=+911.667834895" Nov 02 17:44:53.510237 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:53.510200 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:44:53.510556 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:53.510252 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" podUID="4db49741-b199-403e-98f4-fbba98d95b84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 02 17:44:53.577870 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:53.576232 4860 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 02 17:44:53.577870 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:53.576293 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" podUID="49146668-6dbb-4df5-a1c1-b2c150a74979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 02 17:44:54.019015 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:54.018972 4860 generic.go:356] "Generic (PLEG): container finished" podID="20e25244-8eaa-42e5-8b9f-3324fee7a288" containerID="4009e24fe1459c92af99d12e102e62d2f088033bd3d3e5c0a6c22db77a12d42f" exitCode=0 Nov 02 17:44:54.019415 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:54.019389 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs" event={"ID":"20e25244-8eaa-42e5-8b9f-3324fee7a288","Type":"ContainerDied","Data":"4009e24fe1459c92af99d12e102e62d2f088033bd3d3e5c0a6c22db77a12d42f"} Nov 02 17:44:54.510012 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:54.509977 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:44:54.510422 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:54.510034 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" podUID="4db49741-b199-403e-98f4-fbba98d95b84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 02 17:44:54.575874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:54.575233 4860 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 02 17:44:54.575874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:54.575312 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" podUID="49146668-6dbb-4df5-a1c1-b2c150a74979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 02 17:44:55.350248 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:55.350221 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs" Nov 02 17:44:55.452504 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:55.452464 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20e25244-8eaa-42e5-8b9f-3324fee7a288-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e25244-8eaa-42e5-8b9f-3324fee7a288-maas-api-xks-ca\") pod \"20e25244-8eaa-42e5-8b9f-3324fee7a288\" (UID: \"20e25244-8eaa-42e5-8b9f-3324fee7a288\") " Nov 02 17:44:55.452665 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:55.452529 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20e25244-8eaa-42e5-8b9f-3324fee7a288-kube-api-access-9hlf7\" (UniqueName: \"kubernetes.io/projected/20e25244-8eaa-42e5-8b9f-3324fee7a288-kube-api-access-9hlf7\") pod \"20e25244-8eaa-42e5-8b9f-3324fee7a288\" (UID: \"20e25244-8eaa-42e5-8b9f-3324fee7a288\") " Nov 02 17:44:55.452712 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:55.452695 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20e25244-8eaa-42e5-8b9f-3324fee7a288-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e25244-8eaa-42e5-8b9f-3324fee7a288-maas-api-openshift-ca\") pod \"20e25244-8eaa-42e5-8b9f-3324fee7a288\" (UID: \"20e25244-8eaa-42e5-8b9f-3324fee7a288\") " Nov 02 17:44:55.453379 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:55.453349 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20e25244-8eaa-42e5-8b9f-3324fee7a288-maas-api-openshift-ca" pod "20e25244-8eaa-42e5-8b9f-3324fee7a288" (UID: "20e25244-8eaa-42e5-8b9f-3324fee7a288"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:44:55.511178 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:55.511109 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20e25244-8eaa-42e5-8b9f-3324fee7a288-maas-api-xks-ca" pod "20e25244-8eaa-42e5-8b9f-3324fee7a288" (UID: "20e25244-8eaa-42e5-8b9f-3324fee7a288"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:44:55.512926 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:55.512833 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e25244-8eaa-42e5-8b9f-3324fee7a288-kube-api-access-9hlf7" pod "20e25244-8eaa-42e5-8b9f-3324fee7a288" (UID: "20e25244-8eaa-42e5-8b9f-3324fee7a288"). InnerVolumeSpecName "kube-api-access-9hlf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:55.514515 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:55.514485 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:44:55.514669 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:55.514623 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" podUID="4db49741-b199-403e-98f4-fbba98d95b84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 02 17:44:55.587940 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:55.554814 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e25244-8eaa-42e5-8b9f-3324fee7a288-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:55.587940 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:55.554869 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e25244-8eaa-42e5-8b9f-3324fee7a288-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:55.587940 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:55.554886 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hlf7\" (UniqueName: \"kubernetes.io/projected/20e25244-8eaa-42e5-8b9f-3324fee7a288-kube-api-access-9hlf7\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:44:55.587940 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:55.575209 4860 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 02 17:44:55.587940 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:55.575267 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" podUID="49146668-6dbb-4df5-a1c1-b2c150a74979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 02 17:44:56.039278 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:56.039192 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs" event={"ID":"20e25244-8eaa-42e5-8b9f-3324fee7a288","Type":"ContainerDied","Data":"31d848efcda71c34c23ba3419fc4b55fd7a3d486dfd342b1371479969fd207f0"} Nov 02 17:44:56.039278 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:56.039243 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31d848efcda71c34c23ba3419fc4b55fd7a3d486dfd342b1371479969fd207f0" Nov 02 17:44:56.039278 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:56.039206 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs" Nov 02 17:44:56.511288 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:56.511256 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:44:56.511756 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:56.511730 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" podUID="4db49741-b199-403e-98f4-fbba98d95b84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 02 17:44:56.574873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:56.569263 4860 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 02 17:44:56.574873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:56.569327 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" podUID="49146668-6dbb-4df5-a1c1-b2c150a74979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 02 17:44:57.509736 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:57.509701 4860 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:44:57.509971 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:57.509753 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" podUID="4db49741-b199-403e-98f4-fbba98d95b84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 02 17:44:57.581079 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:57.581028 4860 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 02 17:44:57.581718 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:57.581670 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" podUID="49146668-6dbb-4df5-a1c1-b2c150a74979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 02 17:44:58.568721 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:58.568683 4860 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 02 17:44:58.568948 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:58.568757 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" podUID="49146668-6dbb-4df5-a1c1-b2c150a74979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 02 17:44:58.754443 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:58.754405 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:58.755430 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:58.755404 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:58.756477 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:58.756455 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:44:59.573955 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:59.573927 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:59.574424 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:59.574403 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:44:59.575149 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:44:59.575120 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:45:00.188631 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.188600 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp"] Nov 02 17:45:00.190944 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.190908 4860 cpu_manager.go:414] "containerMap: removing container" podUID="0ebc34e1-d76e-42e6-8f14-a29d41a93f5c" containerName="revoke-keys" Nov 02 17:45:00.190944 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.190944 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebc34e1-d76e-42e6-8f14-a29d41a93f5c" containerName="revoke-keys" Nov 02 17:45:00.191177 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.191105 4860 cpu_manager.go:414] "containerMap: removing container" podUID="20e25244-8eaa-42e5-8b9f-3324fee7a288" containerName="revoke-keys" Nov 02 17:45:00.191177 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.191115 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20e25244-8eaa-42e5-8b9f-3324fee7a288" containerName="revoke-keys" Nov 02 17:45:00.191483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.191467 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="20e25244-8eaa-42e5-8b9f-3324fee7a288" containerName="revoke-keys" Nov 02 17:45:00.191537 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.191497 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebc34e1-d76e-42e6-8f14-a29d41a93f5c" containerName="revoke-keys" Nov 02 17:45:00.198148 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.198115 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" Nov 02 17:45:00.210131 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.210103 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn"] Nov 02 17:45:00.215565 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.215544 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" Nov 02 17:45:00.223161 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.223131 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp"] Nov 02 17:45:00.226824 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.226796 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:45:00.234308 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.234282 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb"] Nov 02 17:45:00.239033 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.239009 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" Nov 02 17:45:00.241555 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.241528 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn"] Nov 02 17:45:00.253268 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.251482 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf"] Nov 02 17:45:00.264254 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.263591 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb"] Nov 02 17:45:00.264254 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.263740 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" Nov 02 17:45:00.270320 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.270299 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf"] Nov 02 17:45:00.277302 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.277276 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c587eb05-2bff-46b1-9ab4-1348b2d629e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894025-9ggvp\" (UID: \"c587eb05-2bff-46b1-9ab4-1348b2d629e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" Nov 02 17:45:00.277538 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.277514 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk2fx\" (UniqueName: \"kubernetes.io/projected/c587eb05-2bff-46b1-9ab4-1348b2d629e6-kube-api-access-xk2fx\") pod \"maas-api-key-cleanup-29894025-9ggvp\" (UID: \"c587eb05-2bff-46b1-9ab4-1348b2d629e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" Nov 02 17:45:00.378490 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.378455 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c587eb05-2bff-46b1-9ab4-1348b2d629e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894025-9ggvp\" (UID: \"c587eb05-2bff-46b1-9ab4-1348b2d629e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" Nov 02 17:45:00.378731 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.378525 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb5p7\" (UniqueName: \"kubernetes.io/projected/e1d65818-7958-42ad-9d12-af8b3ecd3333-kube-api-access-sb5p7\") pod \"maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf\" (UID: \"e1d65818-7958-42ad-9d12-af8b3ecd3333\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" Nov 02 17:45:00.378731 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.378582 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjrbm\" (UniqueName: \"kubernetes.io/projected/5c604bd9-9c15-4710-a843-de008bd9bdad-kube-api-access-hjrbm\") pod \"maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb\" (UID: \"5c604bd9-9c15-4710-a843-de008bd9bdad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" Nov 02 17:45:00.378890 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.378759 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk2fx\" (UniqueName: \"kubernetes.io/projected/c587eb05-2bff-46b1-9ab4-1348b2d629e6-kube-api-access-xk2fx\") pod \"maas-api-key-cleanup-29894025-9ggvp\" (UID: \"c587eb05-2bff-46b1-9ab4-1348b2d629e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" Nov 02 17:45:00.378890 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.378835 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rltc6\" (UniqueName: \"kubernetes.io/projected/5971ac71-edb4-4694-b9c5-d45806c4547b-kube-api-access-rltc6\") pod \"maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn\" (UID: \"5971ac71-edb4-4694-b9c5-d45806c4547b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" Nov 02 17:45:00.379007 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.378889 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1d65818-7958-42ad-9d12-af8b3ecd3333-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf\" (UID: \"e1d65818-7958-42ad-9d12-af8b3ecd3333\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" Nov 02 17:45:00.379044 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.379008 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5971ac71-edb4-4694-b9c5-d45806c4547b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn\" (UID: \"5971ac71-edb4-4694-b9c5-d45806c4547b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" Nov 02 17:45:00.379081 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.379045 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c604bd9-9c15-4710-a843-de008bd9bdad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb\" (UID: \"5c604bd9-9c15-4710-a843-de008bd9bdad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" Nov 02 17:45:00.379238 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.379217 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c587eb05-2bff-46b1-9ab4-1348b2d629e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894025-9ggvp\" (UID: \"c587eb05-2bff-46b1-9ab4-1348b2d629e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" Nov 02 17:45:00.406667 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.406639 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk2fx\" (UniqueName: \"kubernetes.io/projected/c587eb05-2bff-46b1-9ab4-1348b2d629e6-kube-api-access-xk2fx\") pod \"maas-api-key-cleanup-29894025-9ggvp\" (UID: \"c587eb05-2bff-46b1-9ab4-1348b2d629e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" Nov 02 17:45:00.480379 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.480299 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb5p7\" (UniqueName: \"kubernetes.io/projected/e1d65818-7958-42ad-9d12-af8b3ecd3333-kube-api-access-sb5p7\") pod \"maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf\" (UID: \"e1d65818-7958-42ad-9d12-af8b3ecd3333\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" Nov 02 17:45:00.480519 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.480381 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjrbm\" (UniqueName: \"kubernetes.io/projected/5c604bd9-9c15-4710-a843-de008bd9bdad-kube-api-access-hjrbm\") pod \"maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb\" (UID: \"5c604bd9-9c15-4710-a843-de008bd9bdad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" Nov 02 17:45:00.480574 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.480557 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rltc6\" (UniqueName: \"kubernetes.io/projected/5971ac71-edb4-4694-b9c5-d45806c4547b-kube-api-access-rltc6\") pod \"maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn\" (UID: \"5971ac71-edb4-4694-b9c5-d45806c4547b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" Nov 02 17:45:00.480629 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.480594 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1d65818-7958-42ad-9d12-af8b3ecd3333-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf\" (UID: \"e1d65818-7958-42ad-9d12-af8b3ecd3333\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" Nov 02 17:45:00.480720 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.480706 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5971ac71-edb4-4694-b9c5-d45806c4547b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn\" (UID: \"5971ac71-edb4-4694-b9c5-d45806c4547b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" Nov 02 17:45:00.480772 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.480758 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c604bd9-9c15-4710-a843-de008bd9bdad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb\" (UID: \"5c604bd9-9c15-4710-a843-de008bd9bdad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" Nov 02 17:45:00.481380 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.481359 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1d65818-7958-42ad-9d12-af8b3ecd3333-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf\" (UID: \"e1d65818-7958-42ad-9d12-af8b3ecd3333\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" Nov 02 17:45:00.481380 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.481370 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c604bd9-9c15-4710-a843-de008bd9bdad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb\" (UID: \"5c604bd9-9c15-4710-a843-de008bd9bdad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" Nov 02 17:45:00.481468 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.481401 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5971ac71-edb4-4694-b9c5-d45806c4547b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn\" (UID: \"5971ac71-edb4-4694-b9c5-d45806c4547b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" Nov 02 17:45:00.504678 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.504652 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rltc6\" (UniqueName: \"kubernetes.io/projected/5971ac71-edb4-4694-b9c5-d45806c4547b-kube-api-access-rltc6\") pod \"maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn\" (UID: \"5971ac71-edb4-4694-b9c5-d45806c4547b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" Nov 02 17:45:00.504817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.504719 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb5p7\" (UniqueName: \"kubernetes.io/projected/e1d65818-7958-42ad-9d12-af8b3ecd3333-kube-api-access-sb5p7\") pod \"maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf\" (UID: \"e1d65818-7958-42ad-9d12-af8b3ecd3333\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" Nov 02 17:45:00.505011 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.504989 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjrbm\" (UniqueName: \"kubernetes.io/projected/5c604bd9-9c15-4710-a843-de008bd9bdad-kube-api-access-hjrbm\") pod \"maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb\" (UID: \"5c604bd9-9c15-4710-a843-de008bd9bdad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" Nov 02 17:45:00.517910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.517881 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" Nov 02 17:45:00.541157 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.541119 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" Nov 02 17:45:00.565360 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.565328 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" Nov 02 17:45:00.584377 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.583225 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" Nov 02 17:45:00.858080 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:45:00.858044 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc587eb05_2bff_46b1_9ab4_1348b2d629e6.slice/crio-50f0e4df14bed08e42e58c7a3f3c4f64ffd9f85ec21be6c6b57f5d5d995d1ea0 WatchSource:0}: Error finding container 50f0e4df14bed08e42e58c7a3f3c4f64ffd9f85ec21be6c6b57f5d5d995d1ea0: Status 404 returned error can't find the container with id 50f0e4df14bed08e42e58c7a3f3c4f64ffd9f85ec21be6c6b57f5d5d995d1ea0 Nov 02 17:45:00.858223 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.858204 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp"] Nov 02 17:45:00.883009 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.882981 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn"] Nov 02 17:45:00.883121 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:45:00.883107 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5971ac71_edb4_4694_b9c5_d45806c4547b.slice/crio-a4aef2a4bb5789b011547197d37cb00633fe3edbaf6dc0614adbc7cc56983946 WatchSource:0}: Error finding container a4aef2a4bb5789b011547197d37cb00633fe3edbaf6dc0614adbc7cc56983946: Status 404 returned error can't find the container with id a4aef2a4bb5789b011547197d37cb00633fe3edbaf6dc0614adbc7cc56983946 Nov 02 17:45:00.950495 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.950446 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb"] Nov 02 17:45:00.950992 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:45:00.950957 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c604bd9_9c15_4710_a843_de008bd9bdad.slice/crio-4f436516de843d5fda0a0c959fa2aa9439ce81d9fb262525de517ddaf465aa3b WatchSource:0}: Error finding container 4f436516de843d5fda0a0c959fa2aa9439ce81d9fb262525de517ddaf465aa3b: Status 404 returned error can't find the container with id 4f436516de843d5fda0a0c959fa2aa9439ce81d9fb262525de517ddaf465aa3b Nov 02 17:45:00.986597 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:00.986565 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf"] Nov 02 17:45:01.058308 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:45:01.058275 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d65818_7958_42ad_9d12_af8b3ecd3333.slice/crio-b1bb9dea8341731950f79cf783416f94e742db419e57d1eb2b620ed56f22e3ae WatchSource:0}: Error finding container b1bb9dea8341731950f79cf783416f94e742db419e57d1eb2b620ed56f22e3ae: Status 404 returned error can't find the container with id b1bb9dea8341731950f79cf783416f94e742db419e57d1eb2b620ed56f22e3ae Nov 02 17:45:01.093448 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:01.093420 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" event={"ID":"e1d65818-7958-42ad-9d12-af8b3ecd3333","Type":"ContainerStarted","Data":"b1bb9dea8341731950f79cf783416f94e742db419e57d1eb2b620ed56f22e3ae"} Nov 02 17:45:01.094782 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:01.094757 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" event={"ID":"5c604bd9-9c15-4710-a843-de008bd9bdad","Type":"ContainerStarted","Data":"4f436516de843d5fda0a0c959fa2aa9439ce81d9fb262525de517ddaf465aa3b"} Nov 02 17:45:01.096072 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:01.096051 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" event={"ID":"5971ac71-edb4-4694-b9c5-d45806c4547b","Type":"ContainerStarted","Data":"a4aef2a4bb5789b011547197d37cb00633fe3edbaf6dc0614adbc7cc56983946"} Nov 02 17:45:01.097245 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:01.097226 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" event={"ID":"c587eb05-2bff-46b1-9ab4-1348b2d629e6","Type":"ContainerStarted","Data":"50f0e4df14bed08e42e58c7a3f3c4f64ffd9f85ec21be6c6b57f5d5d995d1ea0"} Nov 02 17:45:02.107333 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:02.107296 4860 generic.go:356] "Generic (PLEG): container finished" podID="c587eb05-2bff-46b1-9ab4-1348b2d629e6" containerID="e93e5cead10937930662e63dfd223547f428bb6d6618558c9e89d2476600fff2" exitCode=60 Nov 02 17:45:02.107714 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:02.107378 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" event={"ID":"c587eb05-2bff-46b1-9ab4-1348b2d629e6","Type":"ContainerDied","Data":"e93e5cead10937930662e63dfd223547f428bb6d6618558c9e89d2476600fff2"} Nov 02 17:45:02.107774 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:02.107715 4860 scope.go:122] "RemoveContainer" containerID="e93e5cead10937930662e63dfd223547f428bb6d6618558c9e89d2476600fff2" Nov 02 17:45:02.109337 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:02.109312 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" event={"ID":"e1d65818-7958-42ad-9d12-af8b3ecd3333","Type":"ContainerStarted","Data":"83c0f48a1e6673e0aed676ce91c7c3ef954f6f0c91ee016c432924e9f4c756ee"} Nov 02 17:45:02.111603 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:02.111581 4860 generic.go:356] "Generic (PLEG): container finished" podID="5c604bd9-9c15-4710-a843-de008bd9bdad" containerID="939efd04e56111ea12b44b8e124a436c7687bfb026cc242be3c48bfbe7afda1f" exitCode=60 Nov 02 17:45:02.111708 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:02.111692 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" event={"ID":"5c604bd9-9c15-4710-a843-de008bd9bdad","Type":"ContainerDied","Data":"939efd04e56111ea12b44b8e124a436c7687bfb026cc242be3c48bfbe7afda1f"} Nov 02 17:45:02.112020 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:02.111925 4860 scope.go:122] "RemoveContainer" containerID="939efd04e56111ea12b44b8e124a436c7687bfb026cc242be3c48bfbe7afda1f" Nov 02 17:45:02.113352 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:02.113306 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" event={"ID":"5971ac71-edb4-4694-b9c5-d45806c4547b","Type":"ContainerStarted","Data":"44eeab9dc5f8e48824cbe8265db12fcfea9c06d7cae1eee38e95c1043eb17c45"} Nov 02 17:45:02.235471 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:02.235415 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" podStartSLOduration=2.235400653 podStartE2EDuration="2.235400653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:02.231414201 +0000 UTC m=+920.852773980" watchObservedRunningTime="2026-11-02 17:45:02.235400653 +0000 UTC m=+920.856760425" Nov 02 17:45:02.275406 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:02.275356 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" podStartSLOduration=2.275342891 podStartE2EDuration="2.275342891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:02.271520752 +0000 UTC m=+920.892880527" watchObservedRunningTime="2026-11-02 17:45:02.275342891 +0000 UTC m=+920.896702685" Nov 02 17:45:03.128984 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:03.128936 4860 generic.go:356] "Generic (PLEG): container finished" podID="e1d65818-7958-42ad-9d12-af8b3ecd3333" containerID="83c0f48a1e6673e0aed676ce91c7c3ef954f6f0c91ee016c432924e9f4c756ee" exitCode=60 Nov 02 17:45:03.129452 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:03.129276 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" event={"ID":"e1d65818-7958-42ad-9d12-af8b3ecd3333","Type":"ContainerDied","Data":"83c0f48a1e6673e0aed676ce91c7c3ef954f6f0c91ee016c432924e9f4c756ee"} Nov 02 17:45:03.129959 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:03.129934 4860 scope.go:122] "RemoveContainer" containerID="83c0f48a1e6673e0aed676ce91c7c3ef954f6f0c91ee016c432924e9f4c756ee" Nov 02 17:45:03.133290 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:03.133264 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" event={"ID":"5c604bd9-9c15-4710-a843-de008bd9bdad","Type":"ContainerStarted","Data":"8ad75dc9050a59364c28b3f88cf1c82d7fc1d29cd9c8fa6393d04f59a124ffee"} Nov 02 17:45:03.136625 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:03.136595 4860 generic.go:356] "Generic (PLEG): container finished" podID="5971ac71-edb4-4694-b9c5-d45806c4547b" containerID="44eeab9dc5f8e48824cbe8265db12fcfea9c06d7cae1eee38e95c1043eb17c45" exitCode=60 Nov 02 17:45:03.136676 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:03.136629 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" event={"ID":"5971ac71-edb4-4694-b9c5-d45806c4547b","Type":"ContainerDied","Data":"44eeab9dc5f8e48824cbe8265db12fcfea9c06d7cae1eee38e95c1043eb17c45"} Nov 02 17:45:03.136963 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:03.136939 4860 scope.go:122] "RemoveContainer" containerID="44eeab9dc5f8e48824cbe8265db12fcfea9c06d7cae1eee38e95c1043eb17c45" Nov 02 17:45:03.139046 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:03.139018 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" event={"ID":"c587eb05-2bff-46b1-9ab4-1348b2d629e6","Type":"ContainerStarted","Data":"0736900cf404443b16d8579aa32a6cf019c4443b81e85dd24f349f269bfb16fb"} Nov 02 17:45:03.220277 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:03.220214 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" podStartSLOduration=3.220195085 podStartE2EDuration="3.220195085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:03.215796246 +0000 UTC m=+921.837156022" watchObservedRunningTime="2026-11-02 17:45:03.220195085 +0000 UTC m=+921.841554864" Nov 02 17:45:03.317243 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:03.317185 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" podStartSLOduration=3.3171686400000002 podStartE2EDuration="3.31716864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:03.311897911 +0000 UTC m=+921.933257683" watchObservedRunningTime="2026-11-02 17:45:03.31716864 +0000 UTC m=+921.938528416" Nov 02 17:45:04.151187 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:04.151150 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" event={"ID":"5971ac71-edb4-4694-b9c5-d45806c4547b","Type":"ContainerStarted","Data":"df9530e7743c126a653c9c11a725d9e545733478c1a110c21de74bb0effcbc0c"} Nov 02 17:45:04.153496 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:04.153467 4860 generic.go:356] "Generic (PLEG): container finished" podID="c587eb05-2bff-46b1-9ab4-1348b2d629e6" containerID="0736900cf404443b16d8579aa32a6cf019c4443b81e85dd24f349f269bfb16fb" exitCode=60 Nov 02 17:45:04.153625 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:04.153551 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" event={"ID":"c587eb05-2bff-46b1-9ab4-1348b2d629e6","Type":"ContainerDied","Data":"0736900cf404443b16d8579aa32a6cf019c4443b81e85dd24f349f269bfb16fb"} Nov 02 17:45:04.153625 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:04.153608 4860 scope.go:122] "RemoveContainer" containerID="e93e5cead10937930662e63dfd223547f428bb6d6618558c9e89d2476600fff2" Nov 02 17:45:04.153865 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:04.153830 4860 scope.go:122] "RemoveContainer" containerID="0736900cf404443b16d8579aa32a6cf019c4443b81e85dd24f349f269bfb16fb" Nov 02 17:45:04.154229 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:45:04.154202 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894025-9ggvp_odh-ai-gateway-infra(c587eb05-2bff-46b1-9ab4-1348b2d629e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" podUID="c587eb05-2bff-46b1-9ab4-1348b2d629e6" Nov 02 17:45:04.156539 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:04.156516 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" event={"ID":"e1d65818-7958-42ad-9d12-af8b3ecd3333","Type":"ContainerStarted","Data":"2fdadef80b63864dc7c4e7e55456b43e0bbb39bf6cb8330a57f7f50aacaf6e48"} Nov 02 17:45:04.159370 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:04.159347 4860 generic.go:356] "Generic (PLEG): container finished" podID="5c604bd9-9c15-4710-a843-de008bd9bdad" containerID="8ad75dc9050a59364c28b3f88cf1c82d7fc1d29cd9c8fa6393d04f59a124ffee" exitCode=60 Nov 02 17:45:04.159975 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:04.159947 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" event={"ID":"5c604bd9-9c15-4710-a843-de008bd9bdad","Type":"ContainerDied","Data":"8ad75dc9050a59364c28b3f88cf1c82d7fc1d29cd9c8fa6393d04f59a124ffee"} Nov 02 17:45:04.160050 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:04.159997 4860 scope.go:122] "RemoveContainer" containerID="8ad75dc9050a59364c28b3f88cf1c82d7fc1d29cd9c8fa6393d04f59a124ffee" Nov 02 17:45:04.160243 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:45:04.160217 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb_odh-ai-gateway-infra(5c604bd9-9c15-4710-a843-de008bd9bdad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" podUID="5c604bd9-9c15-4710-a843-de008bd9bdad" Nov 02 17:45:04.296013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:04.295989 4860 scope.go:122] "RemoveContainer" containerID="939efd04e56111ea12b44b8e124a436c7687bfb026cc242be3c48bfbe7afda1f" Nov 02 17:45:05.170824 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:05.170785 4860 generic.go:356] "Generic (PLEG): container finished" podID="5971ac71-edb4-4694-b9c5-d45806c4547b" containerID="df9530e7743c126a653c9c11a725d9e545733478c1a110c21de74bb0effcbc0c" exitCode=60 Nov 02 17:45:05.171328 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:05.170890 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" event={"ID":"5971ac71-edb4-4694-b9c5-d45806c4547b","Type":"ContainerDied","Data":"df9530e7743c126a653c9c11a725d9e545733478c1a110c21de74bb0effcbc0c"} Nov 02 17:45:05.171328 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:05.170941 4860 scope.go:122] "RemoveContainer" containerID="44eeab9dc5f8e48824cbe8265db12fcfea9c06d7cae1eee38e95c1043eb17c45" Nov 02 17:45:05.171328 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:05.171140 4860 scope.go:122] "RemoveContainer" containerID="df9530e7743c126a653c9c11a725d9e545733478c1a110c21de74bb0effcbc0c" Nov 02 17:45:05.171526 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:45:05.171499 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn_odh-ai-gateway-infra(5971ac71-edb4-4694-b9c5-d45806c4547b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" podUID="5971ac71-edb4-4694-b9c5-d45806c4547b" Nov 02 17:45:05.173093 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:05.173073 4860 scope.go:122] "RemoveContainer" containerID="0736900cf404443b16d8579aa32a6cf019c4443b81e85dd24f349f269bfb16fb" Nov 02 17:45:05.173454 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:45:05.173428 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894025-9ggvp_odh-ai-gateway-infra(c587eb05-2bff-46b1-9ab4-1348b2d629e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" podUID="c587eb05-2bff-46b1-9ab4-1348b2d629e6" Nov 02 17:45:05.175206 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:05.175186 4860 generic.go:356] "Generic (PLEG): container finished" podID="e1d65818-7958-42ad-9d12-af8b3ecd3333" containerID="2fdadef80b63864dc7c4e7e55456b43e0bbb39bf6cb8330a57f7f50aacaf6e48" exitCode=60 Nov 02 17:45:05.175308 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:05.175282 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" event={"ID":"e1d65818-7958-42ad-9d12-af8b3ecd3333","Type":"ContainerDied","Data":"2fdadef80b63864dc7c4e7e55456b43e0bbb39bf6cb8330a57f7f50aacaf6e48"} Nov 02 17:45:05.175605 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:05.175589 4860 scope.go:122] "RemoveContainer" containerID="2fdadef80b63864dc7c4e7e55456b43e0bbb39bf6cb8330a57f7f50aacaf6e48" Nov 02 17:45:05.175899 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:45:05.175873 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf_odh-ai-gateway-infra(e1d65818-7958-42ad-9d12-af8b3ecd3333)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" podUID="e1d65818-7958-42ad-9d12-af8b3ecd3333" Nov 02 17:45:05.177743 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:05.177728 4860 scope.go:122] "RemoveContainer" containerID="8ad75dc9050a59364c28b3f88cf1c82d7fc1d29cd9c8fa6393d04f59a124ffee" Nov 02 17:45:05.178036 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:45:05.178014 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb_odh-ai-gateway-infra(5c604bd9-9c15-4710-a843-de008bd9bdad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" podUID="5c604bd9-9c15-4710-a843-de008bd9bdad" Nov 02 17:45:05.187285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:05.187267 4860 scope.go:122] "RemoveContainer" containerID="83c0f48a1e6673e0aed676ce91c7c3ef954f6f0c91ee016c432924e9f4c756ee" Nov 02 17:45:06.189189 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:06.189162 4860 scope.go:122] "RemoveContainer" containerID="2fdadef80b63864dc7c4e7e55456b43e0bbb39bf6cb8330a57f7f50aacaf6e48" Nov 02 17:45:06.189597 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:45:06.189527 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf_odh-ai-gateway-infra(e1d65818-7958-42ad-9d12-af8b3ecd3333)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" podUID="e1d65818-7958-42ad-9d12-af8b3ecd3333" Nov 02 17:45:06.191063 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:06.191044 4860 scope.go:122] "RemoveContainer" containerID="df9530e7743c126a653c9c11a725d9e545733478c1a110c21de74bb0effcbc0c" Nov 02 17:45:06.191347 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:45:06.191329 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn_odh-ai-gateway-infra(5971ac71-edb4-4694-b9c5-d45806c4547b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" podUID="5971ac71-edb4-4694-b9c5-d45806c4547b" Nov 02 17:45:14.480658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:14.480617 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l"] Nov 02 17:45:14.488063 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:14.488040 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l" Nov 02 17:45:14.507958 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:14.507924 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l"] Nov 02 17:45:14.511105 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:14.511008 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:45:14.681083 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:14.681043 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4zhh\" (UniqueName: \"kubernetes.io/projected/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-kube-api-access-v4zhh\") pod \"maas-api-revoke-sub-fba7f293-d6w4l\" (UID: \"70a3c054-80b7-4a3c-890d-7e38e1ba30cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l" Nov 02 17:45:14.681083 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:14.681088 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba7f293-d6w4l\" (UID: \"70a3c054-80b7-4a3c-890d-7e38e1ba30cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l" Nov 02 17:45:14.681357 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:14.681333 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba7f293-d6w4l\" (UID: \"70a3c054-80b7-4a3c-890d-7e38e1ba30cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l" Nov 02 17:45:14.782757 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:14.782679 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba7f293-d6w4l\" (UID: \"70a3c054-80b7-4a3c-890d-7e38e1ba30cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l" Nov 02 17:45:14.782981 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:14.782951 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4zhh\" (UniqueName: \"kubernetes.io/projected/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-kube-api-access-v4zhh\") pod \"maas-api-revoke-sub-fba7f293-d6w4l\" (UID: \"70a3c054-80b7-4a3c-890d-7e38e1ba30cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l" Nov 02 17:45:14.783074 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:14.783005 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba7f293-d6w4l\" (UID: \"70a3c054-80b7-4a3c-890d-7e38e1ba30cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l" Nov 02 17:45:14.783739 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:14.783714 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba7f293-d6w4l\" (UID: \"70a3c054-80b7-4a3c-890d-7e38e1ba30cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l" Nov 02 17:45:14.785236 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:14.785215 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba7f293-d6w4l\" (UID: \"70a3c054-80b7-4a3c-890d-7e38e1ba30cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l" Nov 02 17:45:14.807310 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:14.807285 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4zhh\" (UniqueName: \"kubernetes.io/projected/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-kube-api-access-v4zhh\") pod \"maas-api-revoke-sub-fba7f293-d6w4l\" (UID: \"70a3c054-80b7-4a3c-890d-7e38e1ba30cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l" Nov 02 17:45:15.114888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:15.108134 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l" Nov 02 17:45:15.458024 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:45:15.457994 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a3c054_80b7_4a3c_890d_7e38e1ba30cb.slice/crio-e6d69098769748430c55e6a115ed8fb64514beca73b51b5a02acbcd8711b6414 WatchSource:0}: Error finding container e6d69098769748430c55e6a115ed8fb64514beca73b51b5a02acbcd8711b6414: Status 404 returned error can't find the container with id e6d69098769748430c55e6a115ed8fb64514beca73b51b5a02acbcd8711b6414 Nov 02 17:45:15.460426 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:15.460402 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l"] Nov 02 17:45:15.492115 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:15.492091 4860 scope.go:122] "RemoveContainer" containerID="2fdadef80b63864dc7c4e7e55456b43e0bbb39bf6cb8330a57f7f50aacaf6e48" Nov 02 17:45:15.492425 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:15.492280 4860 scope.go:122] "RemoveContainer" containerID="8ad75dc9050a59364c28b3f88cf1c82d7fc1d29cd9c8fa6393d04f59a124ffee" Nov 02 17:45:16.312906 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:16.312763 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l" event={"ID":"70a3c054-80b7-4a3c-890d-7e38e1ba30cb","Type":"ContainerStarted","Data":"d0179fc051b3bb7797894b87db3c72ed0758bab6d09f8280fb01657b1656c58f"} Nov 02 17:45:16.312906 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:16.312844 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l" event={"ID":"70a3c054-80b7-4a3c-890d-7e38e1ba30cb","Type":"ContainerStarted","Data":"e6d69098769748430c55e6a115ed8fb64514beca73b51b5a02acbcd8711b6414"} Nov 02 17:45:16.315788 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:16.315758 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" event={"ID":"e1d65818-7958-42ad-9d12-af8b3ecd3333","Type":"ContainerStarted","Data":"bac32a0c92b23b051c8c130196e6360f02867479490f250c45506ca6ba885971"} Nov 02 17:45:16.318356 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:16.318331 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" event={"ID":"5c604bd9-9c15-4710-a843-de008bd9bdad","Type":"ContainerStarted","Data":"9efc7d54cc06186d5500376b61ae876cc56d58c7d2f4e73364d5cd436ccae7a0"} Nov 02 17:45:16.361631 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:16.361579 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l" podStartSLOduration=2.361564348 podStartE2EDuration="2.361564348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:16.356843701 +0000 UTC m=+934.978203482" watchObservedRunningTime="2026-11-02 17:45:16.361564348 +0000 UTC m=+934.982924124" Nov 02 17:45:16.492586 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:16.492545 4860 scope.go:122] "RemoveContainer" containerID="df9530e7743c126a653c9c11a725d9e545733478c1a110c21de74bb0effcbc0c" Nov 02 17:45:17.330878 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:17.330827 4860 generic.go:356] "Generic (PLEG): container finished" podID="70a3c054-80b7-4a3c-890d-7e38e1ba30cb" containerID="d0179fc051b3bb7797894b87db3c72ed0758bab6d09f8280fb01657b1656c58f" exitCode=0 Nov 02 17:45:17.331085 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:17.330917 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l" event={"ID":"70a3c054-80b7-4a3c-890d-7e38e1ba30cb","Type":"ContainerDied","Data":"d0179fc051b3bb7797894b87db3c72ed0758bab6d09f8280fb01657b1656c58f"} Nov 02 17:45:17.333202 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:17.333171 4860 generic.go:356] "Generic (PLEG): container finished" podID="e1d65818-7958-42ad-9d12-af8b3ecd3333" containerID="bac32a0c92b23b051c8c130196e6360f02867479490f250c45506ca6ba885971" exitCode=60 Nov 02 17:45:17.333326 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:17.333208 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" event={"ID":"e1d65818-7958-42ad-9d12-af8b3ecd3333","Type":"ContainerDied","Data":"bac32a0c92b23b051c8c130196e6360f02867479490f250c45506ca6ba885971"} Nov 02 17:45:17.333326 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:17.333272 4860 scope.go:122] "RemoveContainer" containerID="2fdadef80b63864dc7c4e7e55456b43e0bbb39bf6cb8330a57f7f50aacaf6e48" Nov 02 17:45:17.333707 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:17.333682 4860 scope.go:122] "RemoveContainer" containerID="bac32a0c92b23b051c8c130196e6360f02867479490f250c45506ca6ba885971" Nov 02 17:45:17.334185 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:45:17.334163 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf_odh-ai-gateway-infra(e1d65818-7958-42ad-9d12-af8b3ecd3333)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" podUID="e1d65818-7958-42ad-9d12-af8b3ecd3333" Nov 02 17:45:17.336055 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:17.336031 4860 generic.go:356] "Generic (PLEG): container finished" podID="5c604bd9-9c15-4710-a843-de008bd9bdad" containerID="9efc7d54cc06186d5500376b61ae876cc56d58c7d2f4e73364d5cd436ccae7a0" exitCode=60 Nov 02 17:45:17.336162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:17.336097 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" event={"ID":"5c604bd9-9c15-4710-a843-de008bd9bdad","Type":"ContainerDied","Data":"9efc7d54cc06186d5500376b61ae876cc56d58c7d2f4e73364d5cd436ccae7a0"} Nov 02 17:45:17.336553 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:17.336528 4860 scope.go:122] "RemoveContainer" containerID="9efc7d54cc06186d5500376b61ae876cc56d58c7d2f4e73364d5cd436ccae7a0" Nov 02 17:45:17.336986 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:45:17.336959 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb_odh-ai-gateway-infra(5c604bd9-9c15-4710-a843-de008bd9bdad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" podUID="5c604bd9-9c15-4710-a843-de008bd9bdad" Nov 02 17:45:17.338573 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:17.338545 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" event={"ID":"5971ac71-edb4-4694-b9c5-d45806c4547b","Type":"ContainerStarted","Data":"a1d4a620ddc341ef13f5b6d87695e5fd3e50983b300faf6791a334cedb3b6d8e"} Nov 02 17:45:17.348977 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:17.348957 4860 scope.go:122] "RemoveContainer" containerID="8ad75dc9050a59364c28b3f88cf1c82d7fc1d29cd9c8fa6393d04f59a124ffee" Nov 02 17:45:17.424119 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:17.424083 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb"] Nov 02 17:45:17.482992 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:17.482954 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf"] Nov 02 17:45:17.491806 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:17.491766 4860 scope.go:122] "RemoveContainer" containerID="0736900cf404443b16d8579aa32a6cf019c4443b81e85dd24f349f269bfb16fb" Nov 02 17:45:18.355805 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.355774 4860 generic.go:356] "Generic (PLEG): container finished" podID="5971ac71-edb4-4694-b9c5-d45806c4547b" containerID="a1d4a620ddc341ef13f5b6d87695e5fd3e50983b300faf6791a334cedb3b6d8e" exitCode=60 Nov 02 17:45:18.356247 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.355886 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" event={"ID":"5971ac71-edb4-4694-b9c5-d45806c4547b","Type":"ContainerDied","Data":"a1d4a620ddc341ef13f5b6d87695e5fd3e50983b300faf6791a334cedb3b6d8e"} Nov 02 17:45:18.356247 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.355958 4860 scope.go:122] "RemoveContainer" containerID="df9530e7743c126a653c9c11a725d9e545733478c1a110c21de74bb0effcbc0c" Nov 02 17:45:18.356398 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.356381 4860 scope.go:122] "RemoveContainer" containerID="a1d4a620ddc341ef13f5b6d87695e5fd3e50983b300faf6791a334cedb3b6d8e" Nov 02 17:45:18.356835 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:45:18.356806 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn_odh-ai-gateway-infra(5971ac71-edb4-4694-b9c5-d45806c4547b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" podUID="5971ac71-edb4-4694-b9c5-d45806c4547b" Nov 02 17:45:18.359406 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.359380 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" event={"ID":"c587eb05-2bff-46b1-9ab4-1348b2d629e6","Type":"ContainerStarted","Data":"fb180308dd3aee96ffcb5420fcadf0dcc44645840ead7119f79284f297f22fd5"} Nov 02 17:45:18.573393 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.573360 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn"] Nov 02 17:45:18.677002 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.676976 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l" Nov 02 17:45:18.680682 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:45:18.680652 4860 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc587eb05_2bff_46b1_9ab4_1348b2d629e6.slice/crio-conmon-fb180308dd3aee96ffcb5420fcadf0dcc44645840ead7119f79284f297f22fd5.scope\": RecentStats: unable to find data in memory cache]" Nov 02 17:45:18.735277 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.735223 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" Nov 02 17:45:18.793872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.790239 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-kube-api-access-v4zhh\" (UniqueName: \"kubernetes.io/projected/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-kube-api-access-v4zhh\") pod \"70a3c054-80b7-4a3c-890d-7e38e1ba30cb\" (UID: \"70a3c054-80b7-4a3c-890d-7e38e1ba30cb\") " Nov 02 17:45:18.793872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.790453 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c604bd9-9c15-4710-a843-de008bd9bdad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c604bd9-9c15-4710-a843-de008bd9bdad-maas-api-service-ca\") pod \"5c604bd9-9c15-4710-a843-de008bd9bdad\" (UID: \"5c604bd9-9c15-4710-a843-de008bd9bdad\") " Nov 02 17:45:18.793872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.790510 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-maas-api-xks-ca\") pod \"70a3c054-80b7-4a3c-890d-7e38e1ba30cb\" (UID: \"70a3c054-80b7-4a3c-890d-7e38e1ba30cb\") " Nov 02 17:45:18.793872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.790560 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-maas-api-openshift-ca\") pod \"70a3c054-80b7-4a3c-890d-7e38e1ba30cb\" (UID: \"70a3c054-80b7-4a3c-890d-7e38e1ba30cb\") " Nov 02 17:45:18.793872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.790763 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c604bd9-9c15-4710-a843-de008bd9bdad-kube-api-access-hjrbm\" (UniqueName: \"kubernetes.io/projected/5c604bd9-9c15-4710-a843-de008bd9bdad-kube-api-access-hjrbm\") pod \"5c604bd9-9c15-4710-a843-de008bd9bdad\" (UID: \"5c604bd9-9c15-4710-a843-de008bd9bdad\") " Nov 02 17:45:18.793872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.791706 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c604bd9-9c15-4710-a843-de008bd9bdad-maas-api-service-ca" pod "5c604bd9-9c15-4710-a843-de008bd9bdad" (UID: "5c604bd9-9c15-4710-a843-de008bd9bdad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:18.812899 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.794813 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-maas-api-openshift-ca" pod "70a3c054-80b7-4a3c-890d-7e38e1ba30cb" (UID: "70a3c054-80b7-4a3c-890d-7e38e1ba30cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:18.812899 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.795152 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c604bd9-9c15-4710-a843-de008bd9bdad-maas-api-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:45:18.812899 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.795173 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:45:18.914104 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.911773 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-maas-api-xks-ca" pod "70a3c054-80b7-4a3c-890d-7e38e1ba30cb" (UID: "70a3c054-80b7-4a3c-890d-7e38e1ba30cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:45:18.914104 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.913089 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:45:18.935879 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.931089 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-kube-api-access-v4zhh" pod "70a3c054-80b7-4a3c-890d-7e38e1ba30cb" (UID: "70a3c054-80b7-4a3c-890d-7e38e1ba30cb"). InnerVolumeSpecName "kube-api-access-v4zhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:18.941893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.940135 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c604bd9-9c15-4710-a843-de008bd9bdad-kube-api-access-hjrbm" pod "5c604bd9-9c15-4710-a843-de008bd9bdad" (UID: "5c604bd9-9c15-4710-a843-de008bd9bdad"). InnerVolumeSpecName "kube-api-access-hjrbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:18.945886 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:18.942123 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" Nov 02 17:45:19.015197 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.013738 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1d65818-7958-42ad-9d12-af8b3ecd3333-kube-api-access-sb5p7\" (UniqueName: \"kubernetes.io/projected/e1d65818-7958-42ad-9d12-af8b3ecd3333-kube-api-access-sb5p7\") pod \"e1d65818-7958-42ad-9d12-af8b3ecd3333\" (UID: \"e1d65818-7958-42ad-9d12-af8b3ecd3333\") " Nov 02 17:45:19.015197 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.014176 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1d65818-7958-42ad-9d12-af8b3ecd3333-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1d65818-7958-42ad-9d12-af8b3ecd3333-maas-api-service-ca\") pod \"e1d65818-7958-42ad-9d12-af8b3ecd3333\" (UID: \"e1d65818-7958-42ad-9d12-af8b3ecd3333\") " Nov 02 17:45:19.015197 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.014825 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjrbm\" (UniqueName: \"kubernetes.io/projected/5c604bd9-9c15-4710-a843-de008bd9bdad-kube-api-access-hjrbm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:45:19.015197 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.014867 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4zhh\" (UniqueName: \"kubernetes.io/projected/70a3c054-80b7-4a3c-890d-7e38e1ba30cb-kube-api-access-v4zhh\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:45:19.024911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.015527 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1d65818-7958-42ad-9d12-af8b3ecd3333-maas-api-service-ca" pod "e1d65818-7958-42ad-9d12-af8b3ecd3333" (UID: "e1d65818-7958-42ad-9d12-af8b3ecd3333"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:19.076890 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.063350 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d65818-7958-42ad-9d12-af8b3ecd3333-kube-api-access-sb5p7" pod "e1d65818-7958-42ad-9d12-af8b3ecd3333" (UID: "e1d65818-7958-42ad-9d12-af8b3ecd3333"). InnerVolumeSpecName "kube-api-access-sb5p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:19.142886 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.137344 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1d65818-7958-42ad-9d12-af8b3ecd3333-maas-api-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:45:19.142886 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.137388 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb5p7\" (UniqueName: \"kubernetes.io/projected/e1d65818-7958-42ad-9d12-af8b3ecd3333-kube-api-access-sb5p7\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:45:19.383875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.381542 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" event={"ID":"5c604bd9-9c15-4710-a843-de008bd9bdad","Type":"ContainerDied","Data":"4f436516de843d5fda0a0c959fa2aa9439ce81d9fb262525de517ddaf465aa3b"} Nov 02 17:45:19.383875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.381643 4860 scope.go:122] "RemoveContainer" containerID="9efc7d54cc06186d5500376b61ae876cc56d58c7d2f4e73364d5cd436ccae7a0" Nov 02 17:45:19.383875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.381748 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb" Nov 02 17:45:19.431874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.427592 4860 generic.go:356] "Generic (PLEG): container finished" podID="c587eb05-2bff-46b1-9ab4-1348b2d629e6" containerID="fb180308dd3aee96ffcb5420fcadf0dcc44645840ead7119f79284f297f22fd5" exitCode=60 Nov 02 17:45:19.431874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.427769 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" event={"ID":"c587eb05-2bff-46b1-9ab4-1348b2d629e6","Type":"ContainerDied","Data":"fb180308dd3aee96ffcb5420fcadf0dcc44645840ead7119f79284f297f22fd5"} Nov 02 17:45:19.431874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.429992 4860 scope.go:122] "RemoveContainer" containerID="fb180308dd3aee96ffcb5420fcadf0dcc44645840ead7119f79284f297f22fd5" Nov 02 17:45:19.431874 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:45:19.430413 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894025-9ggvp_odh-ai-gateway-infra(c587eb05-2bff-46b1-9ab4-1348b2d629e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" podUID="c587eb05-2bff-46b1-9ab4-1348b2d629e6" Nov 02 17:45:19.431874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.431215 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l" event={"ID":"70a3c054-80b7-4a3c-890d-7e38e1ba30cb","Type":"ContainerDied","Data":"e6d69098769748430c55e6a115ed8fb64514beca73b51b5a02acbcd8711b6414"} Nov 02 17:45:19.431874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.431260 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6d69098769748430c55e6a115ed8fb64514beca73b51b5a02acbcd8711b6414" Nov 02 17:45:19.431874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.431327 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l" Nov 02 17:45:19.540869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.525056 4860 scope.go:122] "RemoveContainer" containerID="0736900cf404443b16d8579aa32a6cf019c4443b81e85dd24f349f269bfb16fb" Nov 02 17:45:19.572086 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.572052 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" Nov 02 17:45:19.886396 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.885460 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp"] Nov 02 17:45:19.886396 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.885537 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf" event={"ID":"e1d65818-7958-42ad-9d12-af8b3ecd3333","Type":"ContainerDied","Data":"b1bb9dea8341731950f79cf783416f94e742db419e57d1eb2b620ed56f22e3ae"} Nov 02 17:45:19.886396 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:19.885570 4860 scope.go:122] "RemoveContainer" containerID="bac32a0c92b23b051c8c130196e6360f02867479490f250c45506ca6ba885971" Nov 02 17:45:20.120884 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:20.119665 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf"] Nov 02 17:45:20.168873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:20.164297 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894025-jg8vf"] Nov 02 17:45:20.402932 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:20.377042 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb"] Nov 02 17:45:20.687942 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:20.681976 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:20.712918 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:20.690061 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded" Nov 02 17:45:20.712918 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:20.692355 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 02 17:45:20.785882 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:20.761194 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:20.785882 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:20.783131 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:20.831890 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:20.817654 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:20.831890 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:20.818017 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 02 17:45:20.873027 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:20.872920 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894025-r9gmb"] Nov 02 17:45:21.041880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:21.038644 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" Nov 02 17:45:21.069873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:21.050932 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5971ac71-edb4-4694-b9c5-d45806c4547b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5971ac71-edb4-4694-b9c5-d45806c4547b-maas-api-service-ca\") pod \"5971ac71-edb4-4694-b9c5-d45806c4547b\" (UID: \"5971ac71-edb4-4694-b9c5-d45806c4547b\") " Nov 02 17:45:21.069873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:21.051265 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5971ac71-edb4-4694-b9c5-d45806c4547b-kube-api-access-rltc6\" (UniqueName: \"kubernetes.io/projected/5971ac71-edb4-4694-b9c5-d45806c4547b-kube-api-access-rltc6\") pod \"5971ac71-edb4-4694-b9c5-d45806c4547b\" (UID: \"5971ac71-edb4-4694-b9c5-d45806c4547b\") " Nov 02 17:45:21.069873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:21.063116 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5971ac71-edb4-4694-b9c5-d45806c4547b-maas-api-service-ca" pod "5971ac71-edb4-4694-b9c5-d45806c4547b" (UID: "5971ac71-edb4-4694-b9c5-d45806c4547b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:21.159153 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:21.152117 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5971ac71-edb4-4694-b9c5-d45806c4547b-kube-api-access-rltc6" pod "5971ac71-edb4-4694-b9c5-d45806c4547b" (UID: "5971ac71-edb4-4694-b9c5-d45806c4547b"). InnerVolumeSpecName "kube-api-access-rltc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:21.159153 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:21.153339 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rltc6\" (UniqueName: \"kubernetes.io/projected/5971ac71-edb4-4694-b9c5-d45806c4547b-kube-api-access-rltc6\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:45:21.159153 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:21.153365 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5971ac71-edb4-4694-b9c5-d45806c4547b-maas-api-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:45:21.205874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:21.194081 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 02 17:45:21.243892 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:21.219959 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 17:45:21.291944 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:21.291115 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:21.378394 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:21.377248 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:21.715880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:21.715606 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:21.741911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:21.717693 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 02 17:45:21.784881 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:21.784276 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:21.825887 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:21.824917 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:21.846654 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:21.826899 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:21.846654 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:21.840667 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:21.916585 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:21.892737 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:22.195041 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.194985 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:22.221274 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.221207 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:22.335153 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.335088 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:22.426876 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.388492 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c604bd9-9c15-4710-a843-de008bd9bdad" path="/var/lib/kubelet/pods/5c604bd9-9c15-4710-a843-de008bd9bdad/volumes" Nov 02 17:45:22.478217 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.456159 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:22.509666 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.509636 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" Nov 02 17:45:22.510643 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.510608 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1d65818-7958-42ad-9d12-af8b3ecd3333" path="/var/lib/kubelet/pods/e1d65818-7958-42ad-9d12-af8b3ecd3333/volumes" Nov 02 17:45:22.511524 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:45:22.511496 4860 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.014s" Nov 02 17:45:22.511635 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.511536 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn" event={"ID":"5971ac71-edb4-4694-b9c5-d45806c4547b","Type":"ContainerDied","Data":"a4aef2a4bb5789b011547197d37cb00633fe3edbaf6dc0614adbc7cc56983946"} Nov 02 17:45:22.511635 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.511615 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:45:22.511759 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.511654 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:45:22.511811 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.511786 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 17:45:22.511884 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.511822 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:45:22.511884 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.511879 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:45:22.511999 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.511916 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:45:22.511999 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.511951 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:45:22.511999 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.511984 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:45:22.512131 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.512020 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:45:22.512131 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.512055 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:45:22.513583 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.513560 4860 scope.go:122] "RemoveContainer" containerID="a1d4a620ddc341ef13f5b6d87695e5fd3e50983b300faf6791a334cedb3b6d8e" Nov 02 17:45:22.722669 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.681646 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 02 17:45:22.731360 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.692199 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 02 17:45:22.751283 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.730335 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:22.761924 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.751842 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:45:22.869552 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.869430 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 02 17:45:22.960890 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:22.958047 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:23.006163 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:23.006056 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:23.008015 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:23.007975 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 02 17:45:23.028774 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:23.028727 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:23.135172 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:45:23.135134 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:45:23.200305 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:23.200156 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 02 17:45:23.219201 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:23.219154 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" Nov 02 17:45:23.225071 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:23.225014 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:23.352875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:23.349060 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:23.682691 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:23.483616 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 17:45:23.717684 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:23.587063 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c587eb05-2bff-46b1-9ab4-1348b2d629e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c587eb05-2bff-46b1-9ab4-1348b2d629e6-maas-api-service-ca\") pod \"c587eb05-2bff-46b1-9ab4-1348b2d629e6\" (UID: \"c587eb05-2bff-46b1-9ab4-1348b2d629e6\") " Nov 02 17:45:23.718416 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:23.718385 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c587eb05-2bff-46b1-9ab4-1348b2d629e6-kube-api-access-xk2fx\" (UniqueName: \"kubernetes.io/projected/c587eb05-2bff-46b1-9ab4-1348b2d629e6-kube-api-access-xk2fx\") pod \"c587eb05-2bff-46b1-9ab4-1348b2d629e6\" (UID: \"c587eb05-2bff-46b1-9ab4-1348b2d629e6\") " Nov 02 17:45:23.775876 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:23.738495 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:23.805187 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:23.587509 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c587eb05-2bff-46b1-9ab4-1348b2d629e6-maas-api-service-ca" pod "c587eb05-2bff-46b1-9ab4-1348b2d629e6" (UID: "c587eb05-2bff-46b1-9ab4-1348b2d629e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:23.806125 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:23.768122 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 02 17:45:23.806540 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:23.768206 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 02 17:45:23.807308 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:23.768563 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 02 17:45:23.912791 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:23.893697 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c587eb05-2bff-46b1-9ab4-1348b2d629e6-maas-api-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:45:23.988138 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:23.974392 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:23.988138 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:23.976286 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:24.012870 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.003386 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c587eb05-2bff-46b1-9ab4-1348b2d629e6-kube-api-access-xk2fx" pod "c587eb05-2bff-46b1-9ab4-1348b2d629e6" (UID: "c587eb05-2bff-46b1-9ab4-1348b2d629e6"). InnerVolumeSpecName "kube-api-access-xk2fx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:24.026970 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.026922 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:24.036005 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.035964 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:24.037691 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.037661 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 02 17:45:24.107875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.104020 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk2fx\" (UniqueName: \"kubernetes.io/projected/c587eb05-2bff-46b1-9ab4-1348b2d629e6-kube-api-access-xk2fx\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:45:24.221425 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.221329 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" Nov 02 17:45:24.246602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.246545 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:24.271875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.257168 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:24.303897 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.276570 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:45:24.303897 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.276628 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:45:24.303897 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.276662 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:45:24.303897 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.276678 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp" event={"ID":"c587eb05-2bff-46b1-9ab4-1348b2d629e6","Type":"ContainerDied","Data":"50f0e4df14bed08e42e58c7a3f3c4f64ffd9f85ec21be6c6b57f5d5d995d1ea0"} Nov 02 17:45:24.303897 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.276726 4860 scope.go:122] "RemoveContainer" containerID="fb180308dd3aee96ffcb5420fcadf0dcc44645840ead7119f79284f297f22fd5" Nov 02 17:45:24.335886 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.329323 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 17:45:24.335886 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.329388 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 17:45:24.335886 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.329426 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 17:45:24.397080 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.396244 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 17:45:24.560928 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.558311 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 17:45:24.732894 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.732394 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:24.777093 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.777061 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 17:45:24.782961 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.782924 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 17:45:24.784406 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.784382 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 17:45:24.794894 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.791318 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 17:45:24.800758 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.800704 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 17:45:24.818733 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.812490 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 17:45:24.848590 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.848554 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn"] Nov 02 17:45:24.853891 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.853862 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894025-ms7mn"] Nov 02 17:45:24.900013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.899983 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp"] Nov 02 17:45:24.905111 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:24.905086 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894025-9ggvp"] Nov 02 17:45:25.504860 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:25.504819 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5971ac71-edb4-4694-b9c5-d45806c4547b" path="/var/lib/kubelet/pods/5971ac71-edb4-4694-b9c5-d45806c4547b/volumes" Nov 02 17:45:25.505548 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:25.505526 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c587eb05-2bff-46b1-9ab4-1348b2d629e6" path="/var/lib/kubelet/pods/c587eb05-2bff-46b1-9ab4-1348b2d629e6/volumes" Nov 02 17:45:30.451185 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451151 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6"] Nov 02 17:45:30.451817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451804 4860 cpu_manager.go:414] "containerMap: removing container" podUID="5971ac71-edb4-4694-b9c5-d45806c4547b" containerName="cleanup" Nov 02 17:45:30.451894 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451818 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5971ac71-edb4-4694-b9c5-d45806c4547b" containerName="cleanup" Nov 02 17:45:30.451894 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451832 4860 cpu_manager.go:414] "containerMap: removing container" podUID="e1d65818-7958-42ad-9d12-af8b3ecd3333" containerName="cleanup" Nov 02 17:45:30.451894 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451839 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d65818-7958-42ad-9d12-af8b3ecd3333" containerName="cleanup" Nov 02 17:45:30.451894 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451861 4860 cpu_manager.go:414] "containerMap: removing container" podUID="e1d65818-7958-42ad-9d12-af8b3ecd3333" containerName="cleanup" Nov 02 17:45:30.451894 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451867 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d65818-7958-42ad-9d12-af8b3ecd3333" containerName="cleanup" Nov 02 17:45:30.451894 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451878 4860 cpu_manager.go:414] "containerMap: removing container" podUID="5971ac71-edb4-4694-b9c5-d45806c4547b" containerName="cleanup" Nov 02 17:45:30.451894 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451883 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5971ac71-edb4-4694-b9c5-d45806c4547b" containerName="cleanup" Nov 02 17:45:30.451894 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451895 4860 cpu_manager.go:414] "containerMap: removing container" podUID="e1d65818-7958-42ad-9d12-af8b3ecd3333" containerName="cleanup" Nov 02 17:45:30.451894 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451900 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d65818-7958-42ad-9d12-af8b3ecd3333" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451916 4860 cpu_manager.go:414] "containerMap: removing container" podUID="c587eb05-2bff-46b1-9ab4-1348b2d629e6" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451921 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c587eb05-2bff-46b1-9ab4-1348b2d629e6" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451932 4860 cpu_manager.go:414] "containerMap: removing container" podUID="c587eb05-2bff-46b1-9ab4-1348b2d629e6" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451936 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c587eb05-2bff-46b1-9ab4-1348b2d629e6" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451946 4860 cpu_manager.go:414] "containerMap: removing container" podUID="70a3c054-80b7-4a3c-890d-7e38e1ba30cb" containerName="revoke-keys" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451951 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70a3c054-80b7-4a3c-890d-7e38e1ba30cb" containerName="revoke-keys" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451964 4860 cpu_manager.go:414] "containerMap: removing container" podUID="c587eb05-2bff-46b1-9ab4-1348b2d629e6" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451969 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c587eb05-2bff-46b1-9ab4-1348b2d629e6" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451976 4860 cpu_manager.go:414] "containerMap: removing container" podUID="5c604bd9-9c15-4710-a843-de008bd9bdad" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451980 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c604bd9-9c15-4710-a843-de008bd9bdad" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451989 4860 cpu_manager.go:414] "containerMap: removing container" podUID="5c604bd9-9c15-4710-a843-de008bd9bdad" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.451993 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c604bd9-9c15-4710-a843-de008bd9bdad" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.452001 4860 cpu_manager.go:414] "containerMap: removing container" podUID="5c604bd9-9c15-4710-a843-de008bd9bdad" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.452005 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c604bd9-9c15-4710-a843-de008bd9bdad" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.452107 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c587eb05-2bff-46b1-9ab4-1348b2d629e6" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.452114 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c587eb05-2bff-46b1-9ab4-1348b2d629e6" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.452122 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="70a3c054-80b7-4a3c-890d-7e38e1ba30cb" containerName="revoke-keys" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.452129 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c587eb05-2bff-46b1-9ab4-1348b2d629e6" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.452135 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c604bd9-9c15-4710-a843-de008bd9bdad" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.452139 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c604bd9-9c15-4710-a843-de008bd9bdad" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.452145 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="5971ac71-edb4-4694-b9c5-d45806c4547b" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.452152 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="5971ac71-edb4-4694-b9c5-d45806c4547b" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.452157 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d65818-7958-42ad-9d12-af8b3ecd3333" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.452162 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d65818-7958-42ad-9d12-af8b3ecd3333" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.452166 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="5971ac71-edb4-4694-b9c5-d45806c4547b" containerName="cleanup" Nov 02 17:45:30.452162 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.452172 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d65818-7958-42ad-9d12-af8b3ecd3333" containerName="cleanup" Nov 02 17:45:30.455839 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.455820 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6" Nov 02 17:45:30.476197 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.476172 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6"] Nov 02 17:45:30.505193 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.505164 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:45:30.505670 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.505643 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:45:30.528093 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.528036 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b76de4cb-x4vf6\" (UID: \"bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6" Nov 02 17:45:30.528302 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.528133 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz4h9\" (UniqueName: \"kubernetes.io/projected/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-kube-api-access-hz4h9\") pod \"maas-api-revoke-sub-b76de4cb-x4vf6\" (UID: \"bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6" Nov 02 17:45:30.528377 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.528317 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b76de4cb-x4vf6\" (UID: \"bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6" Nov 02 17:45:30.630193 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.630154 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b76de4cb-x4vf6\" (UID: \"bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6" Nov 02 17:45:30.630378 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.630272 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz4h9\" (UniqueName: \"kubernetes.io/projected/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-kube-api-access-hz4h9\") pod \"maas-api-revoke-sub-b76de4cb-x4vf6\" (UID: \"bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6" Nov 02 17:45:30.630520 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.630496 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b76de4cb-x4vf6\" (UID: \"bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6" Nov 02 17:45:30.631221 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.631195 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b76de4cb-x4vf6\" (UID: \"bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6" Nov 02 17:45:30.632653 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.632631 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b76de4cb-x4vf6\" (UID: \"bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6" Nov 02 17:45:30.661009 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.660980 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz4h9\" (UniqueName: \"kubernetes.io/projected/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-kube-api-access-hz4h9\") pod \"maas-api-revoke-sub-b76de4cb-x4vf6\" (UID: \"bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6" Nov 02 17:45:30.772466 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:30.772384 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6" Nov 02 17:45:31.094300 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:45:31.094268 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd80fd3_2e25_45cb_9d0c_7ccfdb99eda8.slice/crio-41b839289a40f1efabdb9fc67f1c98f5deafe67b6bb06485bfab87505a58b83a WatchSource:0}: Error finding container 41b839289a40f1efabdb9fc67f1c98f5deafe67b6bb06485bfab87505a58b83a: Status 404 returned error can't find the container with id 41b839289a40f1efabdb9fc67f1c98f5deafe67b6bb06485bfab87505a58b83a Nov 02 17:45:31.096333 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:31.096303 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6"] Nov 02 17:45:31.299791 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:31.299755 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6" event={"ID":"bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8","Type":"ContainerStarted","Data":"41b839289a40f1efabdb9fc67f1c98f5deafe67b6bb06485bfab87505a58b83a"} Nov 02 17:45:32.400873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:32.399874 4860 generic.go:356] "Generic (PLEG): container finished" podID="bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8" containerID="915c8b85826241947ab75e3268ed6afaef0c443937ff431a12d5b76c77d58b0d" exitCode=0 Nov 02 17:45:32.400873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:32.400208 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6" event={"ID":"bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8","Type":"ContainerDied","Data":"915c8b85826241947ab75e3268ed6afaef0c443937ff431a12d5b76c77d58b0d"} Nov 02 17:45:33.588325 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:33.588301 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6" Nov 02 17:45:33.703108 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:33.703028 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-maas-api-openshift-ca\") pod \"bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8\" (UID: \"bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8\") " Nov 02 17:45:33.703249 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:33.703196 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-maas-api-xks-ca\") pod \"bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8\" (UID: \"bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8\") " Nov 02 17:45:33.703324 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:33.703266 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-kube-api-access-hz4h9\" (UniqueName: \"kubernetes.io/projected/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-kube-api-access-hz4h9\") pod \"bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8\" (UID: \"bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8\") " Nov 02 17:45:33.703383 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:33.703352 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-maas-api-openshift-ca" pod "bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8" (UID: "bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:33.704184 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:33.704156 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:45:33.705384 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:33.705359 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-maas-api-xks-ca" pod "bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8" (UID: "bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:45:33.705467 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:33.705427 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-kube-api-access-hz4h9" pod "bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8" (UID: "bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8"). InnerVolumeSpecName "kube-api-access-hz4h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:33.805844 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:33.805810 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:45:33.805844 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:33.805839 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz4h9\" (UniqueName: \"kubernetes.io/projected/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8-kube-api-access-hz4h9\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:45:34.087925 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.087876 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt"] Nov 02 17:45:34.088611 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.088596 4860 cpu_manager.go:414] "containerMap: removing container" podUID="bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8" containerName="revoke-keys" Nov 02 17:45:34.088656 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.088613 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8" containerName="revoke-keys" Nov 02 17:45:34.088656 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.088637 4860 cpu_manager.go:414] "containerMap: removing container" podUID="5971ac71-edb4-4694-b9c5-d45806c4547b" containerName="cleanup" Nov 02 17:45:34.088656 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.088642 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5971ac71-edb4-4694-b9c5-d45806c4547b" containerName="cleanup" Nov 02 17:45:34.088751 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.088740 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8" containerName="revoke-keys" Nov 02 17:45:34.088751 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.088749 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c604bd9-9c15-4710-a843-de008bd9bdad" containerName="cleanup" Nov 02 17:45:34.092626 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.092604 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" Nov 02 17:45:34.125566 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.125527 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt"] Nov 02 17:45:34.128671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.128518 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-556cd8a1-serving-cert\"" Nov 02 17:45:34.210439 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.210405 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-metrics-certs\") pod \"maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt\" (UID: \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" Nov 02 17:45:34.210600 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.210465 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxp22\" (UniqueName: \"kubernetes.io/projected/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-kube-api-access-nxp22\") pod \"maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt\" (UID: \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" Nov 02 17:45:34.210641 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.210587 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-openshift-service-ca\") pod \"maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt\" (UID: \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" Nov 02 17:45:34.210680 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.210660 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-maas-api-tls\") pod \"maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt\" (UID: \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" Nov 02 17:45:34.313038 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.313001 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-metrics-certs\") pod \"maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt\" (UID: \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" Nov 02 17:45:34.313573 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.313176 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxp22\" (UniqueName: \"kubernetes.io/projected/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-kube-api-access-nxp22\") pod \"maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt\" (UID: \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" Nov 02 17:45:34.313573 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.313238 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-openshift-service-ca\") pod \"maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt\" (UID: \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" Nov 02 17:45:34.313573 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.313279 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-maas-api-tls\") pod \"maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt\" (UID: \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" Nov 02 17:45:34.314336 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.314304 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-openshift-service-ca\") pod \"maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt\" (UID: \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" Nov 02 17:45:34.317183 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.317158 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-maas-api-tls\") pod \"maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt\" (UID: \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" Nov 02 17:45:34.317912 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.317834 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-metrics-certs\") pod \"maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt\" (UID: \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" Nov 02 17:45:34.348151 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.348063 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxp22\" (UniqueName: \"kubernetes.io/projected/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-kube-api-access-nxp22\") pod \"maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt\" (UID: \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" Nov 02 17:45:34.409001 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.408952 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" Nov 02 17:45:34.432336 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.432129 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6" Nov 02 17:45:34.432336 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.432115 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6" event={"ID":"bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8","Type":"ContainerDied","Data":"41b839289a40f1efabdb9fc67f1c98f5deafe67b6bb06485bfab87505a58b83a"} Nov 02 17:45:34.432336 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.432285 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41b839289a40f1efabdb9fc67f1c98f5deafe67b6bb06485bfab87505a58b83a" Nov 02 17:45:34.746715 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:45:34.746680 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb13f8b29_bfcd_4f5f_84f8_9229dbf546b9.slice/crio-b41d5009601de1a5da7e219abf7e765c52491f16e6c4ee515bd7ed8bb0aa8600 WatchSource:0}: Error finding container b41d5009601de1a5da7e219abf7e765c52491f16e6c4ee515bd7ed8bb0aa8600: Status 404 returned error can't find the container with id b41d5009601de1a5da7e219abf7e765c52491f16e6c4ee515bd7ed8bb0aa8600 Nov 02 17:45:34.748602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.748579 4860 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:45:34.748764 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:34.748743 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt"] Nov 02 17:45:35.457532 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:35.457425 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" event={"ID":"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9","Type":"ContainerStarted","Data":"f3b7dee57d53420cb427ae5d9a045fe1c465b8c439f82ea95c3586bbd774a1d5"} Nov 02 17:45:35.457532 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:35.457492 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" event={"ID":"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9","Type":"ContainerStarted","Data":"b41d5009601de1a5da7e219abf7e765c52491f16e6c4ee515bd7ed8bb0aa8600"} Nov 02 17:45:35.457757 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:35.457558 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" Nov 02 17:45:35.508342 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:35.508292 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" podStartSLOduration=1.176451092 podStartE2EDuration="1.508276336s" totalImagesPullingTime="331.825244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:35.507874715 +0000 UTC m=+954.129234487" watchObservedRunningTime="2026-11-02 17:45:35.508276336 +0000 UTC m=+954.129636112" Nov 02 17:45:39.048365 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:39.048329 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts"] Nov 02 17:45:39.060490 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:39.060465 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf07803e-grrts"] Nov 02 17:45:39.501641 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:39.501607 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1def7ef1-2e02-4174-8b07-e418879e3cdc" path="/var/lib/kubelet/pods/1def7ef1-2e02-4174-8b07-e418879e3cdc/volumes" Nov 02 17:45:41.472917 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:41.472881 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" Nov 02 17:45:46.059308 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:46.059267 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml"] Nov 02 17:45:46.070974 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:46.070944 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf84112-9wlml"] Nov 02 17:45:47.500860 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:47.500811 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b0417d-52a1-4c27-941d-da74c9d5de0d" path="/var/lib/kubelet/pods/c7b0417d-52a1-4c27-941d-da74c9d5de0d/volumes" Nov 02 17:45:51.776311 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:45:51.776264 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:45:59.708440 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:59.708406 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w"] Nov 02 17:45:59.712983 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:59.712960 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w" Nov 02 17:45:59.733120 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:59.733085 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmbr6\" (UniqueName: \"kubernetes.io/projected/661ef687-95f7-4aa9-86a3-b3227f6bf367-kube-api-access-gmbr6\") pod \"maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w\" (UID: \"661ef687-95f7-4aa9-86a3-b3227f6bf367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w" Nov 02 17:45:59.733408 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:59.733384 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661ef687-95f7-4aa9-86a3-b3227f6bf367-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w\" (UID: \"661ef687-95f7-4aa9-86a3-b3227f6bf367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w" Nov 02 17:45:59.733687 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:59.733668 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661ef687-95f7-4aa9-86a3-b3227f6bf367-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w\" (UID: \"661ef687-95f7-4aa9-86a3-b3227f6bf367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w" Nov 02 17:45:59.735379 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:59.735341 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w"] Nov 02 17:45:59.738331 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:59.738304 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:45:59.738472 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:59.738403 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:45:59.834784 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:59.834745 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661ef687-95f7-4aa9-86a3-b3227f6bf367-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w\" (UID: \"661ef687-95f7-4aa9-86a3-b3227f6bf367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w" Nov 02 17:45:59.835111 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:59.835080 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmbr6\" (UniqueName: \"kubernetes.io/projected/661ef687-95f7-4aa9-86a3-b3227f6bf367-kube-api-access-gmbr6\") pod \"maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w\" (UID: \"661ef687-95f7-4aa9-86a3-b3227f6bf367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w" Nov 02 17:45:59.835264 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:59.835127 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661ef687-95f7-4aa9-86a3-b3227f6bf367-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w\" (UID: \"661ef687-95f7-4aa9-86a3-b3227f6bf367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w" Nov 02 17:45:59.835652 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:59.835620 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661ef687-95f7-4aa9-86a3-b3227f6bf367-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w\" (UID: \"661ef687-95f7-4aa9-86a3-b3227f6bf367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w" Nov 02 17:45:59.837820 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:59.837795 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661ef687-95f7-4aa9-86a3-b3227f6bf367-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w\" (UID: \"661ef687-95f7-4aa9-86a3-b3227f6bf367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w" Nov 02 17:45:59.859913 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:45:59.859893 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmbr6\" (UniqueName: \"kubernetes.io/projected/661ef687-95f7-4aa9-86a3-b3227f6bf367-kube-api-access-gmbr6\") pod \"maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w\" (UID: \"661ef687-95f7-4aa9-86a3-b3227f6bf367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w" Nov 02 17:46:00.036022 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:00.035932 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w" Nov 02 17:46:00.364829 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:00.363375 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w"] Nov 02 17:46:00.720697 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:00.720661 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w" event={"ID":"661ef687-95f7-4aa9-86a3-b3227f6bf367","Type":"ContainerStarted","Data":"2ae7f11da07eb9ebb3678b90453f5700b253932fbc0f682ac98088cd8146ddc9"} Nov 02 17:46:00.721101 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:00.720716 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w" event={"ID":"661ef687-95f7-4aa9-86a3-b3227f6bf367","Type":"ContainerStarted","Data":"91a159a21646a24fe72c75b67279bea2d73a8acaebaf263502d1d1c265e5208c"} Nov 02 17:46:00.766929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:00.766876 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w" podStartSLOduration=1.7668449480000001 podStartE2EDuration="1.766844948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:00.762501347 +0000 UTC m=+979.383861131" watchObservedRunningTime="2026-11-02 17:46:00.766844948 +0000 UTC m=+979.388204723" Nov 02 17:46:01.053485 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:01.053392 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l"] Nov 02 17:46:01.059039 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:01.059001 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60a57f8-5gg4l"] Nov 02 17:46:01.502788 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:01.502751 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b0c5a95-b8a3-44f2-a11f-11978c76a712" path="/var/lib/kubelet/pods/7b0c5a95-b8a3-44f2-a11f-11978c76a712/volumes" Nov 02 17:46:01.733499 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:01.733412 4860 generic.go:356] "Generic (PLEG): container finished" podID="661ef687-95f7-4aa9-86a3-b3227f6bf367" containerID="2ae7f11da07eb9ebb3678b90453f5700b253932fbc0f682ac98088cd8146ddc9" exitCode=0 Nov 02 17:46:01.733929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:01.733490 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w" event={"ID":"661ef687-95f7-4aa9-86a3-b3227f6bf367","Type":"ContainerDied","Data":"2ae7f11da07eb9ebb3678b90453f5700b253932fbc0f682ac98088cd8146ddc9"} Nov 02 17:46:02.920388 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:02.920365 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w" Nov 02 17:46:02.984495 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:02.984457 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/661ef687-95f7-4aa9-86a3-b3227f6bf367-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661ef687-95f7-4aa9-86a3-b3227f6bf367-maas-api-xks-ca\") pod \"661ef687-95f7-4aa9-86a3-b3227f6bf367\" (UID: \"661ef687-95f7-4aa9-86a3-b3227f6bf367\") " Nov 02 17:46:02.985422 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:02.984590 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/661ef687-95f7-4aa9-86a3-b3227f6bf367-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661ef687-95f7-4aa9-86a3-b3227f6bf367-maas-api-openshift-ca\") pod \"661ef687-95f7-4aa9-86a3-b3227f6bf367\" (UID: \"661ef687-95f7-4aa9-86a3-b3227f6bf367\") " Nov 02 17:46:02.985422 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:02.984689 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661ef687-95f7-4aa9-86a3-b3227f6bf367-kube-api-access-gmbr6\" (UniqueName: \"kubernetes.io/projected/661ef687-95f7-4aa9-86a3-b3227f6bf367-kube-api-access-gmbr6\") pod \"661ef687-95f7-4aa9-86a3-b3227f6bf367\" (UID: \"661ef687-95f7-4aa9-86a3-b3227f6bf367\") " Nov 02 17:46:02.985422 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:02.985072 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661ef687-95f7-4aa9-86a3-b3227f6bf367-maas-api-openshift-ca" pod "661ef687-95f7-4aa9-86a3-b3227f6bf367" (UID: "661ef687-95f7-4aa9-86a3-b3227f6bf367"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:46:02.985637 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:02.985445 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661ef687-95f7-4aa9-86a3-b3227f6bf367-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:46:02.987317 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:02.987290 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661ef687-95f7-4aa9-86a3-b3227f6bf367-kube-api-access-gmbr6" pod "661ef687-95f7-4aa9-86a3-b3227f6bf367" (UID: "661ef687-95f7-4aa9-86a3-b3227f6bf367"). InnerVolumeSpecName "kube-api-access-gmbr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:46:02.987317 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:02.987304 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/661ef687-95f7-4aa9-86a3-b3227f6bf367-maas-api-xks-ca" pod "661ef687-95f7-4aa9-86a3-b3227f6bf367" (UID: "661ef687-95f7-4aa9-86a3-b3227f6bf367"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:46:03.060213 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:03.060134 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn"] Nov 02 17:46:03.066328 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:03.066298 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69152f6e-l28nn"] Nov 02 17:46:03.086820 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:03.086700 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661ef687-95f7-4aa9-86a3-b3227f6bf367-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:46:03.087108 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:03.087085 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmbr6\" (UniqueName: \"kubernetes.io/projected/661ef687-95f7-4aa9-86a3-b3227f6bf367-kube-api-access-gmbr6\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:46:03.500942 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:03.500909 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da516c96-b934-48dd-9806-24795e13e854" path="/var/lib/kubelet/pods/da516c96-b934-48dd-9806-24795e13e854/volumes" Nov 02 17:46:03.753256 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:03.753171 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w" Nov 02 17:46:03.753256 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:03.753202 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w" event={"ID":"661ef687-95f7-4aa9-86a3-b3227f6bf367","Type":"ContainerDied","Data":"91a159a21646a24fe72c75b67279bea2d73a8acaebaf263502d1d1c265e5208c"} Nov 02 17:46:03.753452 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:03.753262 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91a159a21646a24fe72c75b67279bea2d73a8acaebaf263502d1d1c265e5208c" Nov 02 17:46:04.950235 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:04.950202 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772"] Nov 02 17:46:04.951204 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:04.951185 4860 cpu_manager.go:414] "containerMap: removing container" podUID="661ef687-95f7-4aa9-86a3-b3227f6bf367" containerName="revoke-keys" Nov 02 17:46:04.951204 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:04.951205 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661ef687-95f7-4aa9-86a3-b3227f6bf367" containerName="revoke-keys" Nov 02 17:46:04.951371 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:04.951362 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="661ef687-95f7-4aa9-86a3-b3227f6bf367" containerName="revoke-keys" Nov 02 17:46:04.955651 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:04.955631 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:04.986371 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:04.986332 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772"] Nov 02 17:46:05.019827 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.019277 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94044c36-9d36-41a2-801c-afb684fdfd3c-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.019827 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.019345 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94044c36-9d36-41a2-801c-afb684fdfd3c-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.021545 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.021512 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94044c36-9d36-41a2-801c-afb684fdfd3c-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.021674 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.021578 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94044c36-9d36-41a2-801c-afb684fdfd3c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.021674 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.021607 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65svp\" (UniqueName: \"kubernetes.io/projected/94044c36-9d36-41a2-801c-afb684fdfd3c-kube-api-access-65svp\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.021674 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.021668 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94044c36-9d36-41a2-801c-afb684fdfd3c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.021950 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.021763 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94044c36-9d36-41a2-801c-afb684fdfd3c-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.021950 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.021828 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94044c36-9d36-41a2-801c-afb684fdfd3c-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.022082 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.021985 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94044c36-9d36-41a2-801c-afb684fdfd3c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.048977 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.048950 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l"] Nov 02 17:46:05.054239 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.054214 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d38be61d-7rb5l"] Nov 02 17:46:05.123191 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.123152 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94044c36-9d36-41a2-801c-afb684fdfd3c-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.123358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.123203 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94044c36-9d36-41a2-801c-afb684fdfd3c-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.123358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.123292 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94044c36-9d36-41a2-801c-afb684fdfd3c-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.123358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.123334 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94044c36-9d36-41a2-801c-afb684fdfd3c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.123499 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.123356 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65svp\" (UniqueName: \"kubernetes.io/projected/94044c36-9d36-41a2-801c-afb684fdfd3c-kube-api-access-65svp\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.123499 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.123409 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94044c36-9d36-41a2-801c-afb684fdfd3c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.123499 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.123490 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94044c36-9d36-41a2-801c-afb684fdfd3c-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.123640 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.123547 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94044c36-9d36-41a2-801c-afb684fdfd3c-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.123640 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.123591 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94044c36-9d36-41a2-801c-afb684fdfd3c-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.123739 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.123653 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94044c36-9d36-41a2-801c-afb684fdfd3c-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.123739 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.123678 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94044c36-9d36-41a2-801c-afb684fdfd3c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.123739 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.123691 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94044c36-9d36-41a2-801c-afb684fdfd3c-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.124050 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.124027 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94044c36-9d36-41a2-801c-afb684fdfd3c-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.124377 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.124355 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94044c36-9d36-41a2-801c-afb684fdfd3c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.126229 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.126208 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94044c36-9d36-41a2-801c-afb684fdfd3c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.126402 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.126379 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94044c36-9d36-41a2-801c-afb684fdfd3c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.147527 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.147503 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94044c36-9d36-41a2-801c-afb684fdfd3c-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.147651 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.147632 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65svp\" (UniqueName: \"kubernetes.io/projected/94044c36-9d36-41a2-801c-afb684fdfd3c-kube-api-access-65svp\") pod \"maas-default-gateway-openshift-default-76f4876667-w4772\" (UID: \"94044c36-9d36-41a2-801c-afb684fdfd3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.274585 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.274502 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:05.510800 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.502112 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="547cae52-11eb-4166-b466-6cf0528e2ce2" path="/var/lib/kubelet/pods/547cae52-11eb-4166-b466-6cf0528e2ce2/volumes" Nov 02 17:46:05.599938 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:46:05.599908 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94044c36_9d36_41a2_801c_afb684fdfd3c.slice/crio-7b8d9d88997608e6e59785e7c2e26a3e8b018fbd974be2378a5125a5d9790821 WatchSource:0}: Error finding container 7b8d9d88997608e6e59785e7c2e26a3e8b018fbd974be2378a5125a5d9790821: Status 404 returned error can't find the container with id 7b8d9d88997608e6e59785e7c2e26a3e8b018fbd974be2378a5125a5d9790821 Nov 02 17:46:05.601637 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.601612 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772"] Nov 02 17:46:05.602451 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.602418 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:46:05.602514 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.602501 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:46:05.602568 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.602543 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:46:05.774156 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.774113 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" event={"ID":"94044c36-9d36-41a2-801c-afb684fdfd3c","Type":"ContainerStarted","Data":"697be2009fe7dff5dea800abf9a8e362bd35912865a418cc10297aeaea02f02e"} Nov 02 17:46:05.774321 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.774181 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" event={"ID":"94044c36-9d36-41a2-801c-afb684fdfd3c","Type":"ContainerStarted","Data":"7b8d9d88997608e6e59785e7c2e26a3e8b018fbd974be2378a5125a5d9790821"} Nov 02 17:46:05.843522 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:05.843470 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" podStartSLOduration=1.843455683 podStartE2EDuration="1.843455683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:05.838553896 +0000 UTC m=+984.459913671" watchObservedRunningTime="2026-11-02 17:46:05.843455683 +0000 UTC m=+984.464815461" Nov 02 17:46:06.275874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:06.275354 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:06.278873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:06.278046 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-w4772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 17:46:06.278873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:06.278107 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" podUID="94044c36-9d36-41a2-801c-afb684fdfd3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 17:46:07.279319 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:07.279271 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-w4772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 17:46:07.279905 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:07.279324 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" podUID="94044c36-9d36-41a2-801c-afb684fdfd3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 17:46:08.275873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:08.275293 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-w4772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 17:46:08.275873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:08.275360 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" podUID="94044c36-9d36-41a2-801c-afb684fdfd3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 17:46:09.282871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:09.282541 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-w4772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 17:46:09.282871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:09.282616 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" podUID="94044c36-9d36-41a2-801c-afb684fdfd3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 17:46:10.043535 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:10.043498 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd"] Nov 02 17:46:10.053455 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:10.053424 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50a8b3e8-h2scd"] Nov 02 17:46:10.275330 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:10.275289 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-w4772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 17:46:10.275605 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:10.275576 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" podUID="94044c36-9d36-41a2-801c-afb684fdfd3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 17:46:11.280876 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:11.280283 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-w4772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 17:46:11.280876 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:11.280350 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" podUID="94044c36-9d36-41a2-801c-afb684fdfd3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 17:46:11.502750 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:11.502721 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfb74ceb-3aea-4cb9-b668-9d92404de76c" path="/var/lib/kubelet/pods/dfb74ceb-3aea-4cb9-b668-9d92404de76c/volumes" Nov 02 17:46:12.275668 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:12.275632 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-w4772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 17:46:12.275920 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:12.275684 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" podUID="94044c36-9d36-41a2-801c-afb684fdfd3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 17:46:13.276071 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:13.276020 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-w4772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 17:46:13.276606 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:13.276089 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" podUID="94044c36-9d36-41a2-801c-afb684fdfd3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 17:46:14.061524 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:14.061484 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp"] Nov 02 17:46:14.071050 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:14.071009 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8705900e-q8dbp"] Nov 02 17:46:14.512819 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:14.512723 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:14.513345 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:14.513166 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:14.515882 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:14.515818 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" Nov 02 17:46:15.349093 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.349053 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt"] Nov 02 17:46:15.349657 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.349605 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" podUID="b13f8b29-bfcd-4f5f-84f8-9229dbf546b9" containerName="maas-api" containerID="cri-o://f3b7dee57d53420cb427ae5d9a045fe1c465b8c439f82ea95c3586bbd774a1d5" gracePeriod=30 Nov 02 17:46:15.506069 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.506042 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b19eed-d6d7-4680-b5fc-9ab7e750c851" path="/var/lib/kubelet/pods/40b19eed-d6d7-4680-b5fc-9ab7e750c851/volumes" Nov 02 17:46:15.792103 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.792070 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" Nov 02 17:46:15.909301 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.909262 4860 generic.go:356] "Generic (PLEG): container finished" podID="b13f8b29-bfcd-4f5f-84f8-9229dbf546b9" containerID="f3b7dee57d53420cb427ae5d9a045fe1c465b8c439f82ea95c3586bbd774a1d5" exitCode=0 Nov 02 17:46:15.909474 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.909306 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" Nov 02 17:46:15.909474 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.909353 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" event={"ID":"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9","Type":"ContainerDied","Data":"f3b7dee57d53420cb427ae5d9a045fe1c465b8c439f82ea95c3586bbd774a1d5"} Nov 02 17:46:15.909474 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.909398 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt" event={"ID":"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9","Type":"ContainerDied","Data":"b41d5009601de1a5da7e219abf7e765c52491f16e6c4ee515bd7ed8bb0aa8600"} Nov 02 17:46:15.909474 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.909426 4860 scope.go:122] "RemoveContainer" containerID="f3b7dee57d53420cb427ae5d9a045fe1c465b8c439f82ea95c3586bbd774a1d5" Nov 02 17:46:15.923096 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.923072 4860 scope.go:122] "RemoveContainer" containerID="f3b7dee57d53420cb427ae5d9a045fe1c465b8c439f82ea95c3586bbd774a1d5" Nov 02 17:46:15.925020 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:46:15.924988 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3b7dee57d53420cb427ae5d9a045fe1c465b8c439f82ea95c3586bbd774a1d5\": container with ID starting with f3b7dee57d53420cb427ae5d9a045fe1c465b8c439f82ea95c3586bbd774a1d5 not found: ID does not exist" containerID="f3b7dee57d53420cb427ae5d9a045fe1c465b8c439f82ea95c3586bbd774a1d5" Nov 02 17:46:15.925256 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.925232 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b7dee57d53420cb427ae5d9a045fe1c465b8c439f82ea95c3586bbd774a1d5"} err="failed to get container status \"f3b7dee57d53420cb427ae5d9a045fe1c465b8c439f82ea95c3586bbd774a1d5\": rpc error: code = NotFound desc = could not find container \"f3b7dee57d53420cb427ae5d9a045fe1c465b8c439f82ea95c3586bbd774a1d5\": container with ID starting with f3b7dee57d53420cb427ae5d9a045fe1c465b8c439f82ea95c3586bbd774a1d5 not found: ID does not exist" Nov 02 17:46:15.925948 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.925929 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-metrics-certs\") pod \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\" (UID: \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\") " Nov 02 17:46:15.926936 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.926918 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-kube-api-access-nxp22\" (UniqueName: \"kubernetes.io/projected/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-kube-api-access-nxp22\") pod \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\" (UID: \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\") " Nov 02 17:46:15.927116 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.927099 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-openshift-service-ca\") pod \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\" (UID: \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\") " Nov 02 17:46:15.927283 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.927267 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-maas-api-tls\") pod \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\" (UID: \"b13f8b29-bfcd-4f5f-84f8-9229dbf546b9\") " Nov 02 17:46:15.927770 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.927741 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-openshift-service-ca" pod "b13f8b29-bfcd-4f5f-84f8-9229dbf546b9" (UID: "b13f8b29-bfcd-4f5f-84f8-9229dbf546b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:46:15.928321 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.928300 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:46:15.929314 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.929294 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-metrics-certs" pod "b13f8b29-bfcd-4f5f-84f8-9229dbf546b9" (UID: "b13f8b29-bfcd-4f5f-84f8-9229dbf546b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:46:15.930464 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.930441 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-kube-api-access-nxp22" pod "b13f8b29-bfcd-4f5f-84f8-9229dbf546b9" (UID: "b13f8b29-bfcd-4f5f-84f8-9229dbf546b9"). InnerVolumeSpecName "kube-api-access-nxp22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:46:15.930947 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:15.930922 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-maas-api-tls" pod "b13f8b29-bfcd-4f5f-84f8-9229dbf546b9" (UID: "b13f8b29-bfcd-4f5f-84f8-9229dbf546b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:46:16.032385 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:16.032294 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:46:16.033830 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:16.032621 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:46:16.033830 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:16.032646 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxp22\" (UniqueName: \"kubernetes.io/projected/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9-kube-api-access-nxp22\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:46:16.304438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:16.304412 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w"] Nov 02 17:46:16.311943 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:16.311915 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-556cd8a1-pg24w"] Nov 02 17:46:16.316914 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:16.316837 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt"] Nov 02 17:46:16.322156 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:16.322130 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-556cd8a1-575c4667c8-5t5pt"] Nov 02 17:46:17.054385 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:17.054352 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq"] Nov 02 17:46:17.066512 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:17.066480 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb3a84d1-mt7tq"] Nov 02 17:46:17.502948 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:17.502867 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="661ef687-95f7-4aa9-86a3-b3227f6bf367" path="/var/lib/kubelet/pods/661ef687-95f7-4aa9-86a3-b3227f6bf367/volumes" Nov 02 17:46:17.503400 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:17.503378 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f32aa8-a2fd-4f77-bca4-671f33875f4a" path="/var/lib/kubelet/pods/73f32aa8-a2fd-4f77-bca4-671f33875f4a/volumes" Nov 02 17:46:17.503818 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:17.503801 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b13f8b29-bfcd-4f5f-84f8-9229dbf546b9" path="/var/lib/kubelet/pods/b13f8b29-bfcd-4f5f-84f8-9229dbf546b9/volumes" Nov 02 17:46:20.020977 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.020938 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k"] Nov 02 17:46:20.021720 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.021701 4860 cpu_manager.go:414] "containerMap: removing container" podUID="b13f8b29-bfcd-4f5f-84f8-9229dbf546b9" containerName="maas-api" Nov 02 17:46:20.021720 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.021720 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b13f8b29-bfcd-4f5f-84f8-9229dbf546b9" containerName="maas-api" Nov 02 17:46:20.021900 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.021844 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b13f8b29-bfcd-4f5f-84f8-9229dbf546b9" containerName="maas-api" Nov 02 17:46:20.027675 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.027651 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.060116 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.060082 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k"] Nov 02 17:46:20.152438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.152402 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d5947c-9808-42a0-8a29-55b3590819d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.152602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.152489 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9vp8\" (UniqueName: \"kubernetes.io/projected/68d5947c-9808-42a0-8a29-55b3590819d1-kube-api-access-j9vp8\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.152642 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.152612 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d5947c-9808-42a0-8a29-55b3590819d1-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.152683 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.152668 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d5947c-9808-42a0-8a29-55b3590819d1-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.152813 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.152791 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d5947c-9808-42a0-8a29-55b3590819d1-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.152920 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.152862 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d5947c-9808-42a0-8a29-55b3590819d1-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.152984 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.152968 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d5947c-9808-42a0-8a29-55b3590819d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.153039 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.153018 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d5947c-9808-42a0-8a29-55b3590819d1-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.153116 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.153100 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d5947c-9808-42a0-8a29-55b3590819d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.255063 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.255025 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d5947c-9808-42a0-8a29-55b3590819d1-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.255282 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.255076 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d5947c-9808-42a0-8a29-55b3590819d1-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.255282 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.255116 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d5947c-9808-42a0-8a29-55b3590819d1-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.255282 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.255158 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d5947c-9808-42a0-8a29-55b3590819d1-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.255282 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.255240 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d5947c-9808-42a0-8a29-55b3590819d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.255500 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.255304 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d5947c-9808-42a0-8a29-55b3590819d1-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.255500 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.255343 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d5947c-9808-42a0-8a29-55b3590819d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.255500 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.255455 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d5947c-9808-42a0-8a29-55b3590819d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.255659 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.255520 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d5947c-9808-42a0-8a29-55b3590819d1-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.255659 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.255563 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vp8\" (UniqueName: \"kubernetes.io/projected/68d5947c-9808-42a0-8a29-55b3590819d1-kube-api-access-j9vp8\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.255659 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.255615 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d5947c-9808-42a0-8a29-55b3590819d1-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.255956 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.255931 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d5947c-9808-42a0-8a29-55b3590819d1-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.256161 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.256138 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d5947c-9808-42a0-8a29-55b3590819d1-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.256309 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.256286 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d5947c-9808-42a0-8a29-55b3590819d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.258284 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.258251 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d5947c-9808-42a0-8a29-55b3590819d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.258396 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.258301 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d5947c-9808-42a0-8a29-55b3590819d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.284361 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.284292 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d5947c-9808-42a0-8a29-55b3590819d1-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.285451 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.285416 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9vp8\" (UniqueName: \"kubernetes.io/projected/68d5947c-9808-42a0-8a29-55b3590819d1-kube-api-access-j9vp8\") pod \"maas-default-gateway-openshift-default-76f4876667-2pq2k\" (UID: \"68d5947c-9808-42a0-8a29-55b3590819d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.387429 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.387393 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:20.728105 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.728073 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k"] Nov 02 17:46:20.728531 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.728493 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:46:20.728615 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.728564 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:46:20.728615 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.728594 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:46:20.967343 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.967306 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" event={"ID":"68d5947c-9808-42a0-8a29-55b3590819d1","Type":"ContainerStarted","Data":"db6e75db192644b24901a79035931d788282e44e6c3a9e24576d6460d801c91c"} Nov 02 17:46:20.967491 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:20.967368 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" event={"ID":"68d5947c-9808-42a0-8a29-55b3590819d1","Type":"ContainerStarted","Data":"cc7ee53f8fb3432f7a875d5bb7fcdc4b7aa26e762e5e66995d8265e481f9a23b"} Nov 02 17:46:21.031126 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:21.030995 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" podStartSLOduration=2.030976528 podStartE2EDuration="2.030976528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:21.029287004 +0000 UTC m=+999.650646781" watchObservedRunningTime="2026-11-02 17:46:21.030976528 +0000 UTC m=+999.652336309" Nov 02 17:46:21.049900 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:21.049821 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4"] Nov 02 17:46:21.056243 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:21.056213 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1958d0f3-gxpt4"] Nov 02 17:46:21.414877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:21.406555 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:21.414877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:21.406741 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-2pq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 17:46:21.414877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:21.406788 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" podUID="68d5947c-9808-42a0-8a29-55b3590819d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 17:46:21.502812 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:21.502777 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cad1749-01a6-48b5-91b2-55b83107716c" path="/var/lib/kubelet/pods/0cad1749-01a6-48b5-91b2-55b83107716c/volumes" Nov 02 17:46:21.792083 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:46:21.791976 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:46:22.387965 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:22.387929 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-2pq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 17:46:22.388465 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:22.387991 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" podUID="68d5947c-9808-42a0-8a29-55b3590819d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 17:46:23.388716 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:23.388682 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-2pq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 17:46:23.389152 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:23.388736 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" podUID="68d5947c-9808-42a0-8a29-55b3590819d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 17:46:24.387981 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:24.387946 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-2pq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 17:46:24.388155 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:24.388011 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" podUID="68d5947c-9808-42a0-8a29-55b3590819d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 17:46:25.391519 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:25.388712 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-2pq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 17:46:25.391519 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:25.388792 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" podUID="68d5947c-9808-42a0-8a29-55b3590819d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 17:46:26.035801 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:26.035766 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v"] Nov 02 17:46:26.047398 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:26.047366 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b018f2-h4t4v"] Nov 02 17:46:26.388113 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:26.388081 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-2pq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 17:46:26.388268 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:26.388145 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" podUID="68d5947c-9808-42a0-8a29-55b3590819d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 17:46:27.388519 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:27.388481 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-2pq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 17:46:27.388992 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:27.388555 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" podUID="68d5947c-9808-42a0-8a29-55b3590819d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 17:46:27.503737 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:27.503695 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63da40ed-b7f0-458c-9e88-aa8dae1461a5" path="/var/lib/kubelet/pods/63da40ed-b7f0-458c-9e88-aa8dae1461a5/volumes" Nov 02 17:46:28.050276 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:28.050235 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj"] Nov 02 17:46:28.062132 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:28.062099 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12488095-87xgj"] Nov 02 17:46:28.388740 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:28.388705 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-2pq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 17:46:28.389108 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:28.388767 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" podUID="68d5947c-9808-42a0-8a29-55b3590819d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 17:46:29.510938 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:29.510904 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cff84bf9-4916-469a-ba35-048a68ae6c92" path="/var/lib/kubelet/pods/cff84bf9-4916-469a-ba35-048a68ae6c92/volumes" Nov 02 17:46:29.928059 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:29.927941 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:29.928657 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:29.928633 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:29.930044 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:29.930001 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" Nov 02 17:46:34.068874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:34.066900 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8"] Nov 02 17:46:34.093556 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:34.093516 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19497ce5-j7ln8"] Nov 02 17:46:35.502961 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:35.502919 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edf4ab94-d6ad-460b-a8bb-681833072410" path="/var/lib/kubelet/pods/edf4ab94-d6ad-460b-a8bb-681833072410/volumes" Nov 02 17:46:38.268793 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:38.268759 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2"] Nov 02 17:46:38.276661 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:38.276629 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" Nov 02 17:46:38.298830 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:38.298800 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2"] Nov 02 17:46:38.303679 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:38.303651 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ea104436-serving-cert\"" Nov 02 17:46:38.345890 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:38.345837 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-maas-api-tls\") pod \"maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2\" (UID: \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" Nov 02 17:46:38.346089 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:38.346037 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-openshift-service-ca\") pod \"maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2\" (UID: \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" Nov 02 17:46:38.346216 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:38.346191 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-metrics-certs\") pod \"maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2\" (UID: \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" Nov 02 17:46:38.346289 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:38.346247 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5mbr\" (UniqueName: \"kubernetes.io/projected/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-kube-api-access-x5mbr\") pod \"maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2\" (UID: \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" Nov 02 17:46:38.447400 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:38.447360 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5mbr\" (UniqueName: \"kubernetes.io/projected/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-kube-api-access-x5mbr\") pod \"maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2\" (UID: \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" Nov 02 17:46:38.447640 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:38.447445 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-maas-api-tls\") pod \"maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2\" (UID: \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" Nov 02 17:46:38.447640 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:38.447587 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-openshift-service-ca\") pod \"maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2\" (UID: \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" Nov 02 17:46:38.447795 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:38.447738 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-metrics-certs\") pod \"maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2\" (UID: \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" Nov 02 17:46:38.448840 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:38.448810 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-openshift-service-ca\") pod \"maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2\" (UID: \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" Nov 02 17:46:38.450763 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:38.450731 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-metrics-certs\") pod \"maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2\" (UID: \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" Nov 02 17:46:38.452566 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:38.452541 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-maas-api-tls\") pod \"maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2\" (UID: \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" Nov 02 17:46:38.476792 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:38.476758 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5mbr\" (UniqueName: \"kubernetes.io/projected/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-kube-api-access-x5mbr\") pod \"maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2\" (UID: \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" Nov 02 17:46:38.610402 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:38.610360 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" Nov 02 17:46:38.953181 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:46:38.953138 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef74175d_a3d0_4558_8b66_f746a2b9c1f4.slice/crio-04f90245f56e58a5c0b6b7b183fa6d587010b608823a9e3db0b631e83055a751 WatchSource:0}: Error finding container 04f90245f56e58a5c0b6b7b183fa6d587010b608823a9e3db0b631e83055a751: Status 404 returned error can't find the container with id 04f90245f56e58a5c0b6b7b183fa6d587010b608823a9e3db0b631e83055a751 Nov 02 17:46:38.954543 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:38.954516 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2"] Nov 02 17:46:39.429626 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:39.429588 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" event={"ID":"ef74175d-a3d0-4558-8b66-f746a2b9c1f4","Type":"ContainerStarted","Data":"21f1c9b5cdb56a4491e718d7abf3dc958df40480135bbd6b8552179b5c82db01"} Nov 02 17:46:39.430231 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:39.430194 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" event={"ID":"ef74175d-a3d0-4558-8b66-f746a2b9c1f4","Type":"ContainerStarted","Data":"04f90245f56e58a5c0b6b7b183fa6d587010b608823a9e3db0b631e83055a751"} Nov 02 17:46:39.430365 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:39.430241 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" Nov 02 17:46:41.718363 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:41.718331 4860 scope.go:122] "RemoveContainer" containerID="f063160d7576a291bccc11fa9832ea699738dfabfd13cb5abe5856813fd04500" Nov 02 17:46:41.894385 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:41.894324 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" podStartSLOduration=3.5690286970000002 podStartE2EDuration="3.894307528s" totalImagesPullingTime="325.278831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:39.485067852 +0000 UTC m=+1018.106427629" watchObservedRunningTime="2026-11-02 17:46:41.894307528 +0000 UTC m=+1020.515667304" Nov 02 17:46:41.898750 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:41.898725 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9"] Nov 02 17:46:41.904685 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:41.904665 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" Nov 02 17:46:41.926051 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:41.926016 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9"] Nov 02 17:46:41.932032 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:41.932002 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-93321a4f-serving-cert\"" Nov 02 17:46:42.025124 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:42.025035 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97793ae3-47b5-45ef-86c1-73c0fec966a1-metrics-certs\") pod \"maas-api-e2e-derive-93321a4f-59794c8777-qtfm9\" (UID: \"97793ae3-47b5-45ef-86c1-73c0fec966a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" Nov 02 17:46:42.025124 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:42.025079 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97793ae3-47b5-45ef-86c1-73c0fec966a1-maas-api-tls\") pod \"maas-api-e2e-derive-93321a4f-59794c8777-qtfm9\" (UID: \"97793ae3-47b5-45ef-86c1-73c0fec966a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" Nov 02 17:46:42.025336 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:42.025301 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97793ae3-47b5-45ef-86c1-73c0fec966a1-openshift-service-ca\") pod \"maas-api-e2e-derive-93321a4f-59794c8777-qtfm9\" (UID: \"97793ae3-47b5-45ef-86c1-73c0fec966a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" Nov 02 17:46:42.025381 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:42.025337 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb9cl\" (UniqueName: \"kubernetes.io/projected/97793ae3-47b5-45ef-86c1-73c0fec966a1-kube-api-access-bb9cl\") pod \"maas-api-e2e-derive-93321a4f-59794c8777-qtfm9\" (UID: \"97793ae3-47b5-45ef-86c1-73c0fec966a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" Nov 02 17:46:42.043939 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:42.043907 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n"] Nov 02 17:46:42.052806 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:42.052773 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b664d769-mw68n"] Nov 02 17:46:42.127105 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:42.127048 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97793ae3-47b5-45ef-86c1-73c0fec966a1-openshift-service-ca\") pod \"maas-api-e2e-derive-93321a4f-59794c8777-qtfm9\" (UID: \"97793ae3-47b5-45ef-86c1-73c0fec966a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" Nov 02 17:46:42.127674 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:42.127125 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb9cl\" (UniqueName: \"kubernetes.io/projected/97793ae3-47b5-45ef-86c1-73c0fec966a1-kube-api-access-bb9cl\") pod \"maas-api-e2e-derive-93321a4f-59794c8777-qtfm9\" (UID: \"97793ae3-47b5-45ef-86c1-73c0fec966a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" Nov 02 17:46:42.127674 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:42.127292 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97793ae3-47b5-45ef-86c1-73c0fec966a1-metrics-certs\") pod \"maas-api-e2e-derive-93321a4f-59794c8777-qtfm9\" (UID: \"97793ae3-47b5-45ef-86c1-73c0fec966a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" Nov 02 17:46:42.127674 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:42.127325 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97793ae3-47b5-45ef-86c1-73c0fec966a1-maas-api-tls\") pod \"maas-api-e2e-derive-93321a4f-59794c8777-qtfm9\" (UID: \"97793ae3-47b5-45ef-86c1-73c0fec966a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" Nov 02 17:46:42.128056 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:42.127900 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97793ae3-47b5-45ef-86c1-73c0fec966a1-openshift-service-ca\") pod \"maas-api-e2e-derive-93321a4f-59794c8777-qtfm9\" (UID: \"97793ae3-47b5-45ef-86c1-73c0fec966a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" Nov 02 17:46:42.130734 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:42.130699 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97793ae3-47b5-45ef-86c1-73c0fec966a1-metrics-certs\") pod \"maas-api-e2e-derive-93321a4f-59794c8777-qtfm9\" (UID: \"97793ae3-47b5-45ef-86c1-73c0fec966a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" Nov 02 17:46:42.135837 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:42.135792 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97793ae3-47b5-45ef-86c1-73c0fec966a1-maas-api-tls\") pod \"maas-api-e2e-derive-93321a4f-59794c8777-qtfm9\" (UID: \"97793ae3-47b5-45ef-86c1-73c0fec966a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" Nov 02 17:46:42.151659 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:42.151628 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb9cl\" (UniqueName: \"kubernetes.io/projected/97793ae3-47b5-45ef-86c1-73c0fec966a1-kube-api-access-bb9cl\") pod \"maas-api-e2e-derive-93321a4f-59794c8777-qtfm9\" (UID: \"97793ae3-47b5-45ef-86c1-73c0fec966a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" Nov 02 17:46:42.220009 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:42.219975 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" Nov 02 17:46:42.545419 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:46:42.545363 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97793ae3_47b5_45ef_86c1_73c0fec966a1.slice/crio-e1fba6e20bccef98b4aedceb2745a10c9ad850b0081ecdce9c0a0af21e30322d WatchSource:0}: Error finding container e1fba6e20bccef98b4aedceb2745a10c9ad850b0081ecdce9c0a0af21e30322d: Status 404 returned error can't find the container with id e1fba6e20bccef98b4aedceb2745a10c9ad850b0081ecdce9c0a0af21e30322d Nov 02 17:46:42.546721 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:42.546697 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9"] Nov 02 17:46:43.470236 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:43.470195 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" event={"ID":"97793ae3-47b5-45ef-86c1-73c0fec966a1","Type":"ContainerStarted","Data":"a55e57035b7ed89f90ff6e9104f0fe77e82156fc34371a6752f91d7d3722b99c"} Nov 02 17:46:43.470730 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:43.470263 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" event={"ID":"97793ae3-47b5-45ef-86c1-73c0fec966a1","Type":"ContainerStarted","Data":"e1fba6e20bccef98b4aedceb2745a10c9ad850b0081ecdce9c0a0af21e30322d"} Nov 02 17:46:43.470730 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:43.470294 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" Nov 02 17:46:43.502027 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:43.501989 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="264e4e66-95ca-4b21-ad34-8e4317335082" path="/var/lib/kubelet/pods/264e4e66-95ca-4b21-ad34-8e4317335082/volumes" Nov 02 17:46:43.517022 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:43.516974 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" podStartSLOduration=2.144879337 podStartE2EDuration="2.516960727s" totalImagesPullingTime="372.08139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:43.513658052 +0000 UTC m=+1022.135017828" watchObservedRunningTime="2026-11-02 17:46:43.516960727 +0000 UTC m=+1022.138320502" Nov 02 17:46:44.549543 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.549511 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5"] Nov 02 17:46:44.554435 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.554416 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.569444 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.569421 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 17:46:44.570036 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.570008 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 17:46:44.570251 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.570225 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 17:46:44.582174 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.582147 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dklz6\"" Nov 02 17:46:44.591879 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.591833 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5"] Nov 02 17:46:44.686557 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.686519 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a632f9-0197-4f8c-b414-c15b29caea6e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.686736 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.686657 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.686781 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.686745 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.686883 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.686831 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whsl4\" (UniqueName: \"kubernetes.io/projected/10a632f9-0197-4f8c-b414-c15b29caea6e-kube-api-access-whsl4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.686970 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.686947 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.687078 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.687062 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.687188 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.687175 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.788464 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.788429 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.788643 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.788501 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.788643 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.788551 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whsl4\" (UniqueName: \"kubernetes.io/projected/10a632f9-0197-4f8c-b414-c15b29caea6e-kube-api-access-whsl4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.788643 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.788592 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.788770 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.788651 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.788770 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.788760 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.788901 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.788845 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a632f9-0197-4f8c-b414-c15b29caea6e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.789454 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.789426 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.791158 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.791104 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.791285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.791159 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.791285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.791197 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a632f9-0197-4f8c-b414-c15b29caea6e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.791497 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.791475 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.812590 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.812556 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whsl4\" (UniqueName: \"kubernetes.io/projected/10a632f9-0197-4f8c-b414-c15b29caea6e-kube-api-access-whsl4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.813650 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.813627 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9kkj5\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:44.869512 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:44.869479 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:45.199985 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:46:45.199951 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a632f9_0197_4f8c_b414_c15b29caea6e.slice/crio-c4d7e2519c4f03fecb7d6f58620fbb947b6da733f4bc269fe259e2b2d3d834cd WatchSource:0}: Error finding container c4d7e2519c4f03fecb7d6f58620fbb947b6da733f4bc269fe259e2b2d3d834cd: Status 404 returned error can't find the container with id c4d7e2519c4f03fecb7d6f58620fbb947b6da733f4bc269fe259e2b2d3d834cd Nov 02 17:46:45.201438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:45.201407 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5"] Nov 02 17:46:45.447686 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:45.447608 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" Nov 02 17:46:45.490428 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:45.490377 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" event={"ID":"10a632f9-0197-4f8c-b414-c15b29caea6e","Type":"ContainerStarted","Data":"c4d7e2519c4f03fecb7d6f58620fbb947b6da733f4bc269fe259e2b2d3d834cd"} Nov 02 17:46:49.485978 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:49.485946 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" Nov 02 17:46:51.772368 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:46:51.772328 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:46:55.013824 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:55.013789 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:46:55.014268 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:55.013870 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:46:55.616839 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:55.616683 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" event={"ID":"10a632f9-0197-4f8c-b414-c15b29caea6e","Type":"ContainerStarted","Data":"b2e39f38ce529a4f34acf22b28591824ea70d6c4ad384d6e00d423dfccb5cf5a"} Nov 02 17:46:55.616839 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:55.616788 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:46:55.730954 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:55.729542 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" podStartSLOduration=1.919640957 podStartE2EDuration="11.729525565s" totalImagesPullingTime="9.809884608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:55.676808707 +0000 UTC m=+1034.298168483" watchObservedRunningTime="2026-11-02 17:46:55.729525565 +0000 UTC m=+1034.350885341" Nov 02 17:46:56.629835 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:46:56.629802 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 17:47:19.058760 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.058727 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2"] Nov 02 17:47:19.059505 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.059086 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" podUID="ef74175d-a3d0-4558-8b66-f746a2b9c1f4" containerName="maas-api" containerID="cri-o://21f1c9b5cdb56a4491e718d7abf3dc958df40480135bbd6b8552179b5c82db01" gracePeriod=30 Nov 02 17:47:19.372920 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.345976 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv"] Nov 02 17:47:19.372920 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.356283 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.397387 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.397310 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv"] Nov 02 17:47:19.447950 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.442743 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-93321a4f-openshift-default-dockercfg-8fp78\"" Nov 02 17:47:19.452227 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.451999 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" Nov 02 17:47:19.564870 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.564263 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-data\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.564870 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.564320 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9xxz\" (UniqueName: \"kubernetes.io/projected/e735f624-5f46-4abe-ac1e-e2c6e8029edc-kube-api-access-t9xxz\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.564870 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.564361 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-podinfo\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.564870 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.564394 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istiod-ca-cert\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.564870 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.564437 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-token\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.564870 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.564573 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-workload-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.564870 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.564590 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-workload-certs\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.564870 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.564612 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-envoy\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.564870 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.564663 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-credential-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.665381 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.665262 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-openshift-service-ca\") pod \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\" (UID: \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\") " Nov 02 17:47:19.665595 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.665408 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-kube-api-access-x5mbr\" (UniqueName: \"kubernetes.io/projected/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-kube-api-access-x5mbr\") pod \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\" (UID: \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\") " Nov 02 17:47:19.665595 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.665463 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-maas-api-tls\") pod \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\" (UID: \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\") " Nov 02 17:47:19.665595 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.665547 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-metrics-certs\") pod \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\" (UID: \"ef74175d-a3d0-4558-8b66-f746a2b9c1f4\") " Nov 02 17:47:19.666043 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.666011 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-podinfo\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.666289 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.666266 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istiod-ca-cert\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.666486 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.666467 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-token\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.666797 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.666765 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-workload-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.666950 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.666809 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-workload-certs\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.667981 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.667322 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-workload-certs\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.667981 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.667412 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-envoy\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.667981 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.667522 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-credential-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.667981 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.667594 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-data\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.667981 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.667618 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istiod-ca-cert\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.667981 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.667643 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-workload-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.667981 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.667651 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9xxz\" (UniqueName: \"kubernetes.io/projected/e735f624-5f46-4abe-ac1e-e2c6e8029edc-kube-api-access-t9xxz\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.668428 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.668213 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-credential-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.668955 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.668638 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-data\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.669043 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.669009 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-openshift-service-ca" pod "ef74175d-a3d0-4558-8b66-f746a2b9c1f4" (UID: "ef74175d-a3d0-4558-8b66-f746a2b9c1f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:47:19.679892 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.677407 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-maas-api-tls" pod "ef74175d-a3d0-4558-8b66-f746a2b9c1f4" (UID: "ef74175d-a3d0-4558-8b66-f746a2b9c1f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:47:19.679892 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.677445 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-kube-api-access-x5mbr" pod "ef74175d-a3d0-4558-8b66-f746a2b9c1f4" (UID: "ef74175d-a3d0-4558-8b66-f746a2b9c1f4"). InnerVolumeSpecName "kube-api-access-x5mbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:19.679892 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.677527 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-envoy\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.679892 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.677834 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-podinfo\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.679892 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.678778 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-metrics-certs" pod "ef74175d-a3d0-4558-8b66-f746a2b9c1f4" (UID: "ef74175d-a3d0-4558-8b66-f746a2b9c1f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:47:19.692586 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.692554 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-token\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.693169 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.693145 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9xxz\" (UniqueName: \"kubernetes.io/projected/e735f624-5f46-4abe-ac1e-e2c6e8029edc-kube-api-access-t9xxz\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.709898 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.709870 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:19.792874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.777350 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:47:19.792874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.777390 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5mbr\" (UniqueName: \"kubernetes.io/projected/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-kube-api-access-x5mbr\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:47:19.792874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.777409 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:47:19.792874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.777423 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef74175d-a3d0-4558-8b66-f746a2b9c1f4-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:47:19.924984 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.920401 4860 generic.go:356] "Generic (PLEG): container finished" podID="ef74175d-a3d0-4558-8b66-f746a2b9c1f4" containerID="21f1c9b5cdb56a4491e718d7abf3dc958df40480135bbd6b8552179b5c82db01" exitCode=0 Nov 02 17:47:19.924984 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.920458 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" Nov 02 17:47:19.924984 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.920474 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" event={"ID":"ef74175d-a3d0-4558-8b66-f746a2b9c1f4","Type":"ContainerDied","Data":"21f1c9b5cdb56a4491e718d7abf3dc958df40480135bbd6b8552179b5c82db01"} Nov 02 17:47:19.924984 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.920529 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2" event={"ID":"ef74175d-a3d0-4558-8b66-f746a2b9c1f4","Type":"ContainerDied","Data":"04f90245f56e58a5c0b6b7b183fa6d587010b608823a9e3db0b631e83055a751"} Nov 02 17:47:19.924984 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.920552 4860 scope.go:122] "RemoveContainer" containerID="21f1c9b5cdb56a4491e718d7abf3dc958df40480135bbd6b8552179b5c82db01" Nov 02 17:47:19.947422 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.945573 4860 scope.go:122] "RemoveContainer" containerID="21f1c9b5cdb56a4491e718d7abf3dc958df40480135bbd6b8552179b5c82db01" Nov 02 17:47:19.947422 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:47:19.946023 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21f1c9b5cdb56a4491e718d7abf3dc958df40480135bbd6b8552179b5c82db01\": container with ID starting with 21f1c9b5cdb56a4491e718d7abf3dc958df40480135bbd6b8552179b5c82db01 not found: ID does not exist" containerID="21f1c9b5cdb56a4491e718d7abf3dc958df40480135bbd6b8552179b5c82db01" Nov 02 17:47:19.947422 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.946064 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21f1c9b5cdb56a4491e718d7abf3dc958df40480135bbd6b8552179b5c82db01"} err="failed to get container status \"21f1c9b5cdb56a4491e718d7abf3dc958df40480135bbd6b8552179b5c82db01\": rpc error: code = NotFound desc = could not find container \"21f1c9b5cdb56a4491e718d7abf3dc958df40480135bbd6b8552179b5c82db01\": container with ID starting with 21f1c9b5cdb56a4491e718d7abf3dc958df40480135bbd6b8552179b5c82db01 not found: ID does not exist" Nov 02 17:47:19.987829 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.987787 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2"] Nov 02 17:47:19.996830 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:19.996795 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ea104436-69845d5d4d-px9g2"] Nov 02 17:47:20.146254 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:20.144632 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv"] Nov 02 17:47:20.154118 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:20.154082 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:47:20.154246 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:20.154171 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:47:20.154246 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:20.154218 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:47:20.932928 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:20.932888 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" event={"ID":"e735f624-5f46-4abe-ac1e-e2c6e8029edc","Type":"ContainerStarted","Data":"e7ef7acd04a0af2af2484140885b3df6284d3c9485130160ba88c5ee598d1eaa"} Nov 02 17:47:20.933099 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:20.932943 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" event={"ID":"e735f624-5f46-4abe-ac1e-e2c6e8029edc","Type":"ContainerStarted","Data":"02e8bd12e185266d7c6dd3b617b5f2c3ab7333ca6e281ef8d9627221f96e3894"} Nov 02 17:47:20.980737 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:20.980683 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" podStartSLOduration=1.9806699490000002 podStartE2EDuration="1.980669949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:20.978422334 +0000 UTC m=+1059.599782107" watchObservedRunningTime="2026-11-02 17:47:20.980669949 +0000 UTC m=+1059.602029783" Nov 02 17:47:21.502462 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:21.502426 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef74175d-a3d0-4558-8b66-f746a2b9c1f4" path="/var/lib/kubelet/pods/ef74175d-a3d0-4558-8b66-f746a2b9c1f4/volumes" Nov 02 17:47:21.710339 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:21.710309 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:21.711961 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:21.711928 4860 patch_prober.go:29] interesting pod/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 02 17:47:21.712101 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:21.711993 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" podUID="e735f624-5f46-4abe-ac1e-e2c6e8029edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 02 17:47:21.746695 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:21.746657 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9"] Nov 02 17:47:21.747013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:21.746983 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" podUID="97793ae3-47b5-45ef-86c1-73c0fec966a1" containerName="maas-api" containerID="cri-o://a55e57035b7ed89f90ff6e9104f0fe77e82156fc34371a6752f91d7d3722b99c" gracePeriod=30 Nov 02 17:47:22.233506 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:47:22.233465 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:47:22.644761 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:22.644731 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" Nov 02 17:47:22.718056 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:22.718009 4860 patch_prober.go:29] interesting pod/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 02 17:47:22.718248 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:22.718098 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" podUID="e735f624-5f46-4abe-ac1e-e2c6e8029edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 02 17:47:22.745894 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:22.745422 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97793ae3-47b5-45ef-86c1-73c0fec966a1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97793ae3-47b5-45ef-86c1-73c0fec966a1-maas-api-tls\") pod \"97793ae3-47b5-45ef-86c1-73c0fec966a1\" (UID: \"97793ae3-47b5-45ef-86c1-73c0fec966a1\") " Nov 02 17:47:22.746088 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:22.745979 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97793ae3-47b5-45ef-86c1-73c0fec966a1-kube-api-access-bb9cl\" (UniqueName: \"kubernetes.io/projected/97793ae3-47b5-45ef-86c1-73c0fec966a1-kube-api-access-bb9cl\") pod \"97793ae3-47b5-45ef-86c1-73c0fec966a1\" (UID: \"97793ae3-47b5-45ef-86c1-73c0fec966a1\") " Nov 02 17:47:22.746819 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:22.746777 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97793ae3-47b5-45ef-86c1-73c0fec966a1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97793ae3-47b5-45ef-86c1-73c0fec966a1-openshift-service-ca\") pod \"97793ae3-47b5-45ef-86c1-73c0fec966a1\" (UID: \"97793ae3-47b5-45ef-86c1-73c0fec966a1\") " Nov 02 17:47:22.746972 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:22.746831 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97793ae3-47b5-45ef-86c1-73c0fec966a1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/97793ae3-47b5-45ef-86c1-73c0fec966a1-metrics-certs\") pod \"97793ae3-47b5-45ef-86c1-73c0fec966a1\" (UID: \"97793ae3-47b5-45ef-86c1-73c0fec966a1\") " Nov 02 17:47:22.747330 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:22.747280 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97793ae3-47b5-45ef-86c1-73c0fec966a1-openshift-service-ca" pod "97793ae3-47b5-45ef-86c1-73c0fec966a1" (UID: "97793ae3-47b5-45ef-86c1-73c0fec966a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:47:22.747965 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:22.747930 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97793ae3-47b5-45ef-86c1-73c0fec966a1-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:47:22.749800 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:22.748495 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97793ae3-47b5-45ef-86c1-73c0fec966a1-maas-api-tls" pod "97793ae3-47b5-45ef-86c1-73c0fec966a1" (UID: "97793ae3-47b5-45ef-86c1-73c0fec966a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:47:22.750361 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:22.750323 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97793ae3-47b5-45ef-86c1-73c0fec966a1-metrics-certs" pod "97793ae3-47b5-45ef-86c1-73c0fec966a1" (UID: "97793ae3-47b5-45ef-86c1-73c0fec966a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:47:22.753447 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:22.753405 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97793ae3-47b5-45ef-86c1-73c0fec966a1-kube-api-access-bb9cl" pod "97793ae3-47b5-45ef-86c1-73c0fec966a1" (UID: "97793ae3-47b5-45ef-86c1-73c0fec966a1"). InnerVolumeSpecName "kube-api-access-bb9cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:22.850332 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:22.850217 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97793ae3-47b5-45ef-86c1-73c0fec966a1-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:47:22.850332 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:22.850256 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb9cl\" (UniqueName: \"kubernetes.io/projected/97793ae3-47b5-45ef-86c1-73c0fec966a1-kube-api-access-bb9cl\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:47:22.850332 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:22.850285 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97793ae3-47b5-45ef-86c1-73c0fec966a1-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:47:23.020927 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:23.017188 4860 generic.go:356] "Generic (PLEG): container finished" podID="97793ae3-47b5-45ef-86c1-73c0fec966a1" containerID="a55e57035b7ed89f90ff6e9104f0fe77e82156fc34371a6752f91d7d3722b99c" exitCode=0 Nov 02 17:47:23.020927 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:23.018279 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" Nov 02 17:47:23.021509 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:23.021234 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" event={"ID":"97793ae3-47b5-45ef-86c1-73c0fec966a1","Type":"ContainerDied","Data":"a55e57035b7ed89f90ff6e9104f0fe77e82156fc34371a6752f91d7d3722b99c"} Nov 02 17:47:23.021509 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:23.021362 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9" event={"ID":"97793ae3-47b5-45ef-86c1-73c0fec966a1","Type":"ContainerDied","Data":"e1fba6e20bccef98b4aedceb2745a10c9ad850b0081ecdce9c0a0af21e30322d"} Nov 02 17:47:23.021509 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:23.021402 4860 scope.go:122] "RemoveContainer" containerID="a55e57035b7ed89f90ff6e9104f0fe77e82156fc34371a6752f91d7d3722b99c" Nov 02 17:47:23.122308 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:23.122272 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv"] Nov 02 17:47:23.213881 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:23.212907 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9"] Nov 02 17:47:23.322883 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:23.321713 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93321a4f-59794c8777-qtfm9"] Nov 02 17:47:23.335119 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:23.334709 4860 scope.go:122] "RemoveContainer" containerID="a55e57035b7ed89f90ff6e9104f0fe77e82156fc34371a6752f91d7d3722b99c" Nov 02 17:47:23.340351 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:47:23.340317 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a55e57035b7ed89f90ff6e9104f0fe77e82156fc34371a6752f91d7d3722b99c\": container with ID starting with a55e57035b7ed89f90ff6e9104f0fe77e82156fc34371a6752f91d7d3722b99c not found: ID does not exist" containerID="a55e57035b7ed89f90ff6e9104f0fe77e82156fc34371a6752f91d7d3722b99c" Nov 02 17:47:23.340573 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:23.340543 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a55e57035b7ed89f90ff6e9104f0fe77e82156fc34371a6752f91d7d3722b99c"} err="failed to get container status \"a55e57035b7ed89f90ff6e9104f0fe77e82156fc34371a6752f91d7d3722b99c\": rpc error: code = NotFound desc = could not find container \"a55e57035b7ed89f90ff6e9104f0fe77e82156fc34371a6752f91d7d3722b99c\": container with ID starting with a55e57035b7ed89f90ff6e9104f0fe77e82156fc34371a6752f91d7d3722b99c not found: ID does not exist" Nov 02 17:47:23.634027 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:23.626908 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97793ae3-47b5-45ef-86c1-73c0fec966a1" path="/var/lib/kubelet/pods/97793ae3-47b5-45ef-86c1-73c0fec966a1/volumes" Nov 02 17:47:23.710381 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:23.710342 4860 patch_prober.go:29] interesting pod/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 02 17:47:23.710776 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:23.710420 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" podUID="e735f624-5f46-4abe-ac1e-e2c6e8029edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 02 17:47:24.119882 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:24.119772 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" podUID="e735f624-5f46-4abe-ac1e-e2c6e8029edc" containerName="istio-proxy" containerID="cri-o://e7ef7acd04a0af2af2484140885b3df6284d3c9485130160ba88c5ee598d1eaa" gracePeriod=30 Nov 02 17:47:31.394005 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.393954 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4"] Nov 02 17:47:31.395075 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.395041 4860 cpu_manager.go:414] "containerMap: removing container" podUID="ef74175d-a3d0-4558-8b66-f746a2b9c1f4" containerName="maas-api" Nov 02 17:47:31.395269 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.395246 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef74175d-a3d0-4558-8b66-f746a2b9c1f4" containerName="maas-api" Nov 02 17:47:31.395415 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.395318 4860 cpu_manager.go:414] "containerMap: removing container" podUID="97793ae3-47b5-45ef-86c1-73c0fec966a1" containerName="maas-api" Nov 02 17:47:31.395415 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.395326 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97793ae3-47b5-45ef-86c1-73c0fec966a1" containerName="maas-api" Nov 02 17:47:31.395565 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.395553 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="97793ae3-47b5-45ef-86c1-73c0fec966a1" containerName="maas-api" Nov 02 17:47:31.395634 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.395578 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef74175d-a3d0-4558-8b66-f746a2b9c1f4" containerName="maas-api" Nov 02 17:47:31.416281 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.416253 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.443325 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.443272 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4"] Nov 02 17:47:31.494708 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.494671 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-b91583-openshift-default-dockercfg-68cp5\"" Nov 02 17:47:31.590582 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.590541 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-podinfo\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.590871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.590741 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istiod-ca-cert\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.590871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.590785 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-workload-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.591070 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.590917 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-workload-certs\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.591070 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.590954 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-data\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.592127 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.592101 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-token\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.592223 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.592168 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-envoy\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.599011 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.598972 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5jf4\" (UniqueName: \"kubernetes.io/projected/da05c910-1ef7-47e6-a5bf-ead3b63064d8-kube-api-access-g5jf4\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.600772 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.599089 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-credential-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.701399 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.701310 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istiod-ca-cert\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.701399 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.701352 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-workload-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.701621 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.701404 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-workload-certs\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.701621 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.701421 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-data\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.701621 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.701469 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-token\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.701621 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.701513 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-envoy\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.701621 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.701576 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5jf4\" (UniqueName: \"kubernetes.io/projected/da05c910-1ef7-47e6-a5bf-ead3b63064d8-kube-api-access-g5jf4\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.701621 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.701614 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-credential-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.701898 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.701734 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-podinfo\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.702489 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.702202 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istiod-ca-cert\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.702489 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.702290 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-workload-certs\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.702489 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.702339 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-data\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.702489 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.702382 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-workload-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.702489 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.702451 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-credential-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.704401 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.704371 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-envoy\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.705265 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.705246 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-podinfo\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.727906 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.727007 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-token\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.727906 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.727665 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5jf4\" (UniqueName: \"kubernetes.io/projected/da05c910-1ef7-47e6-a5bf-ead3b63064d8-kube-api-access-g5jf4\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:31.739547 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:31.739521 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:32.131164 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:47:32.131131 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda05c910_1ef7_47e6_a5bf_ead3b63064d8.slice/crio-b8467dc11ce269c295dcde035ff6f9d8041afe580f8b6573ad05adb7b68269a1 WatchSource:0}: Error finding container b8467dc11ce269c295dcde035ff6f9d8041afe580f8b6573ad05adb7b68269a1: Status 404 returned error can't find the container with id b8467dc11ce269c295dcde035ff6f9d8041afe580f8b6573ad05adb7b68269a1 Nov 02 17:47:32.132335 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.132308 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4"] Nov 02 17:47:32.133825 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.133791 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:47:32.133950 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.133882 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:47:32.133950 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.133935 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:47:32.226212 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.226169 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" event={"ID":"da05c910-1ef7-47e6-a5bf-ead3b63064d8","Type":"ContainerStarted","Data":"ce2df665bf6d53a0003f3558f90086f5fc27f9b115b78810cadf5fb7d8533973"} Nov 02 17:47:32.226388 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.226244 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" event={"ID":"da05c910-1ef7-47e6-a5bf-ead3b63064d8","Type":"ContainerStarted","Data":"b8467dc11ce269c295dcde035ff6f9d8041afe580f8b6573ad05adb7b68269a1"} Nov 02 17:47:32.229046 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.229016 4860 generic.go:356] "Generic (PLEG): container finished" podID="e735f624-5f46-4abe-ac1e-e2c6e8029edc" containerID="e7ef7acd04a0af2af2484140885b3df6284d3c9485130160ba88c5ee598d1eaa" exitCode=0 Nov 02 17:47:32.229193 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.229085 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" event={"ID":"e735f624-5f46-4abe-ac1e-e2c6e8029edc","Type":"ContainerDied","Data":"e7ef7acd04a0af2af2484140885b3df6284d3c9485130160ba88c5ee598d1eaa"} Nov 02 17:47:32.271922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.271843 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" podStartSLOduration=1.271828667 podStartE2EDuration="1.271828667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:32.271299093 +0000 UTC m=+1070.892658869" watchObservedRunningTime="2026-11-02 17:47:32.271828667 +0000 UTC m=+1070.893188442" Nov 02 17:47:32.645470 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.645443 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:32.720408 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.720234 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-token\" (UniqueName: \"kubernetes.io/projected/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-token\") pod \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " Nov 02 17:47:32.720408 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.720330 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istiod-ca-cert\") pod \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " Nov 02 17:47:32.720408 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.720373 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-envoy\") pod \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " Nov 02 17:47:32.720693 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.720444 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-data\") pod \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " Nov 02 17:47:32.720693 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.720515 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e735f624-5f46-4abe-ac1e-e2c6e8029edc-kube-api-access-t9xxz\" (UniqueName: \"kubernetes.io/projected/e735f624-5f46-4abe-ac1e-e2c6e8029edc-kube-api-access-t9xxz\") pod \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " Nov 02 17:47:32.720693 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.720579 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-workload-socket\") pod \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " Nov 02 17:47:32.720693 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.720645 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-credential-socket\") pod \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " Nov 02 17:47:32.720693 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.720684 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-podinfo\") pod \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " Nov 02 17:47:32.720920 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.720742 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-workload-certs\") pod \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\" (UID: \"e735f624-5f46-4abe-ac1e-e2c6e8029edc\") " Nov 02 17:47:32.722343 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.721975 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-workload-certs" pod "e735f624-5f46-4abe-ac1e-e2c6e8029edc" (UID: "e735f624-5f46-4abe-ac1e-e2c6e8029edc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:32.722343 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.722298 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istiod-ca-cert" pod "e735f624-5f46-4abe-ac1e-e2c6e8029edc" (UID: "e735f624-5f46-4abe-ac1e-e2c6e8029edc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:47:32.723361 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.723333 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-data" pod "e735f624-5f46-4abe-ac1e-e2c6e8029edc" (UID: "e735f624-5f46-4abe-ac1e-e2c6e8029edc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:32.723465 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.723361 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-credential-socket" pod "e735f624-5f46-4abe-ac1e-e2c6e8029edc" (UID: "e735f624-5f46-4abe-ac1e-e2c6e8029edc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:32.723697 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.723678 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-workload-socket" pod "e735f624-5f46-4abe-ac1e-e2c6e8029edc" (UID: "e735f624-5f46-4abe-ac1e-e2c6e8029edc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:32.723974 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.723949 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-token" pod "e735f624-5f46-4abe-ac1e-e2c6e8029edc" (UID: "e735f624-5f46-4abe-ac1e-e2c6e8029edc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:32.725677 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.725644 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e735f624-5f46-4abe-ac1e-e2c6e8029edc-kube-api-access-t9xxz" pod "e735f624-5f46-4abe-ac1e-e2c6e8029edc" (UID: "e735f624-5f46-4abe-ac1e-e2c6e8029edc"). InnerVolumeSpecName "kube-api-access-t9xxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:32.727472 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.727442 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-podinfo" pod "e735f624-5f46-4abe-ac1e-e2c6e8029edc" (UID: "e735f624-5f46-4abe-ac1e-e2c6e8029edc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:47:32.727705 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.727682 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-envoy" pod "e735f624-5f46-4abe-ac1e-e2c6e8029edc" (UID: "e735f624-5f46-4abe-ac1e-e2c6e8029edc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:47:32.739863 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.739819 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:32.823661 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.823625 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.823661 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.823661 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.823888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.823676 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9xxz\" (UniqueName: \"kubernetes.io/projected/e735f624-5f46-4abe-ac1e-e2c6e8029edc-kube-api-access-t9xxz\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.823888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.823689 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.823888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.823703 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.823888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.823715 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.823888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.823729 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e735f624-5f46-4abe-ac1e-e2c6e8029edc-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.823888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.823744 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:47:32.823888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:32.823755 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e735f624-5f46-4abe-ac1e-e2c6e8029edc-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:47:33.240272 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:33.240196 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" Nov 02 17:47:33.240272 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:33.240225 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv" event={"ID":"e735f624-5f46-4abe-ac1e-e2c6e8029edc","Type":"ContainerDied","Data":"02e8bd12e185266d7c6dd3b617b5f2c3ab7333ca6e281ef8d9627221f96e3894"} Nov 02 17:47:33.240478 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:33.240314 4860 scope.go:122] "RemoveContainer" containerID="e7ef7acd04a0af2af2484140885b3df6284d3c9485130160ba88c5ee598d1eaa" Nov 02 17:47:33.307437 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:33.307403 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv"] Nov 02 17:47:33.318897 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:33.318865 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-g7bzv"] Nov 02 17:47:33.504424 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:33.504343 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e735f624-5f46-4abe-ac1e-e2c6e8029edc" path="/var/lib/kubelet/pods/e735f624-5f46-4abe-ac1e-e2c6e8029edc/volumes" Nov 02 17:47:33.762934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:33.760978 4860 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 02 17:47:33.762934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:33.761048 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" podUID="da05c910-1ef7-47e6-a5bf-ead3b63064d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 02 17:47:34.740423 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:34.740396 4860 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 02 17:47:34.740602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:34.740454 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" podUID="da05c910-1ef7-47e6-a5bf-ead3b63064d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 02 17:47:35.740105 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:35.740071 4860 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 02 17:47:35.740501 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:35.740141 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" podUID="da05c910-1ef7-47e6-a5bf-ead3b63064d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 02 17:47:36.741208 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:36.741155 4860 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 02 17:47:36.741693 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:36.741220 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" podUID="da05c910-1ef7-47e6-a5bf-ead3b63064d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 02 17:47:37.740684 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:37.740643 4860 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 02 17:47:37.740897 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:37.740698 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" podUID="da05c910-1ef7-47e6-a5bf-ead3b63064d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 02 17:47:38.744797 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:38.744767 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:38.745305 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:38.745280 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:38.746140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:38.746116 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 17:47:41.769026 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:41.768994 4860 scope.go:122] "RemoveContainer" containerID="09df0d7a6b0663308be7f8f0cefc3e9c1587d14c152a11b4c72b9c4aff7a242b" Nov 02 17:47:41.780220 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:41.780200 4860 scope.go:122] "RemoveContainer" containerID="88b6743b6a7c6771235d3186ec334b2285eb8ae01f1aa6ec078166140659517c" Nov 02 17:47:41.792203 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:41.792179 4860 scope.go:122] "RemoveContainer" containerID="42b6bde4e19ef7abfd00ed2d747527278faae48eaec3c14c6bb6a019f54ba979" Nov 02 17:47:41.805620 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:41.805588 4860 scope.go:122] "RemoveContainer" containerID="de5b9932f440791d80252d51572b5ba5f6f9dc89a1b84aa86a49323658f7d0a1" Nov 02 17:47:41.820106 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:41.820087 4860 scope.go:122] "RemoveContainer" containerID="38c43e0230b35c0ff04def00c8783931e373341d30086a43ed80bba13077c5c6" Nov 02 17:47:41.831648 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:41.831630 4860 scope.go:122] "RemoveContainer" containerID="d9429a0acf4d452815f182e653e8d2812dbb89322a1f3e30501bd46c6709d839" Nov 02 17:47:41.841389 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:41.841364 4860 scope.go:122] "RemoveContainer" containerID="1baf86abe78610217575dd1dcb525ff5e948e9acc2abfa37af064de7c42a85a0" Nov 02 17:47:41.851009 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:41.850988 4860 scope.go:122] "RemoveContainer" containerID="9fb1c4c845ee9f35a164cd0957604eb057b15fabe327ee45e9398829c3122749" Nov 02 17:47:41.861666 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:41.861649 4860 scope.go:122] "RemoveContainer" containerID="37dac0de009ad39b8c96b873da9167a5a442b3e91f34f6bbda82d046722ff57a" Nov 02 17:47:41.872131 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:41.872109 4860 scope.go:122] "RemoveContainer" containerID="a6d2d9a2613c04904534050d0257ccf348d66ef88ab61aa852183ead28dfd968" Nov 02 17:47:41.881588 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:41.881570 4860 scope.go:122] "RemoveContainer" containerID="86c1e5292175283cb9e2f90496a67606231ecfee1a77a590a1d6a16d25457433" Nov 02 17:47:41.899718 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:41.899535 4860 scope.go:122] "RemoveContainer" containerID="5bb112a88bbfa2679f04631615505d996974439ad4100649abfd3197d3fca812" Nov 02 17:47:51.772203 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:47:51.772164 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:47:52.367862 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.367825 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs"] Nov 02 17:47:52.368569 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.368555 4860 cpu_manager.go:414] "containerMap: removing container" podUID="e735f624-5f46-4abe-ac1e-e2c6e8029edc" containerName="istio-proxy" Nov 02 17:47:52.368625 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.368571 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e735f624-5f46-4abe-ac1e-e2c6e8029edc" containerName="istio-proxy" Nov 02 17:47:52.368709 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.368699 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e735f624-5f46-4abe-ac1e-e2c6e8029edc" containerName="istio-proxy" Nov 02 17:47:52.376343 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.376322 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" Nov 02 17:47:52.392295 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.392272 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs"] Nov 02 17:47:52.401208 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.401187 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-f91b2d-serving-cert\"" Nov 02 17:47:52.416226 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.416199 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/134da1f2-4c24-495f-9b0a-76d21ba04df4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs\" (UID: \"134da1f2-4c24-495f-9b0a-76d21ba04df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" Nov 02 17:47:52.416366 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.416336 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tjv5\" (UniqueName: \"kubernetes.io/projected/134da1f2-4c24-495f-9b0a-76d21ba04df4-kube-api-access-5tjv5\") pod \"maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs\" (UID: \"134da1f2-4c24-495f-9b0a-76d21ba04df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" Nov 02 17:47:52.416509 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.416488 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/134da1f2-4c24-495f-9b0a-76d21ba04df4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs\" (UID: \"134da1f2-4c24-495f-9b0a-76d21ba04df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" Nov 02 17:47:52.416614 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.416597 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/134da1f2-4c24-495f-9b0a-76d21ba04df4-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs\" (UID: \"134da1f2-4c24-495f-9b0a-76d21ba04df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" Nov 02 17:47:52.518116 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.518082 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/134da1f2-4c24-495f-9b0a-76d21ba04df4-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs\" (UID: \"134da1f2-4c24-495f-9b0a-76d21ba04df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" Nov 02 17:47:52.518280 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.518229 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/134da1f2-4c24-495f-9b0a-76d21ba04df4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs\" (UID: \"134da1f2-4c24-495f-9b0a-76d21ba04df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" Nov 02 17:47:52.518342 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.518327 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tjv5\" (UniqueName: \"kubernetes.io/projected/134da1f2-4c24-495f-9b0a-76d21ba04df4-kube-api-access-5tjv5\") pod \"maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs\" (UID: \"134da1f2-4c24-495f-9b0a-76d21ba04df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" Nov 02 17:47:52.518462 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.518447 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/134da1f2-4c24-495f-9b0a-76d21ba04df4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs\" (UID: \"134da1f2-4c24-495f-9b0a-76d21ba04df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" Nov 02 17:47:52.519045 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.519019 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/134da1f2-4c24-495f-9b0a-76d21ba04df4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs\" (UID: \"134da1f2-4c24-495f-9b0a-76d21ba04df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" Nov 02 17:47:52.520657 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.520634 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/134da1f2-4c24-495f-9b0a-76d21ba04df4-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs\" (UID: \"134da1f2-4c24-495f-9b0a-76d21ba04df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" Nov 02 17:47:52.521079 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.521060 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/134da1f2-4c24-495f-9b0a-76d21ba04df4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs\" (UID: \"134da1f2-4c24-495f-9b0a-76d21ba04df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" Nov 02 17:47:52.543120 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.543091 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tjv5\" (UniqueName: \"kubernetes.io/projected/134da1f2-4c24-495f-9b0a-76d21ba04df4-kube-api-access-5tjv5\") pod \"maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs\" (UID: \"134da1f2-4c24-495f-9b0a-76d21ba04df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" Nov 02 17:47:52.692572 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:52.692478 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" Nov 02 17:47:53.020277 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:47:53.020239 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134da1f2_4c24_495f_9b0a_76d21ba04df4.slice/crio-19c0b39c21858081f7ca89acdc091209edcfb1e0a6d204213cc8327e5726ba52 WatchSource:0}: Error finding container 19c0b39c21858081f7ca89acdc091209edcfb1e0a6d204213cc8327e5726ba52: Status 404 returned error can't find the container with id 19c0b39c21858081f7ca89acdc091209edcfb1e0a6d204213cc8327e5726ba52 Nov 02 17:47:53.021790 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:53.021764 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs"] Nov 02 17:47:53.692193 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:53.692088 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" event={"ID":"134da1f2-4c24-495f-9b0a-76d21ba04df4","Type":"ContainerStarted","Data":"5749a8cd039c6ecdfa34638533d303da9163cabb5dca7c79c3b44a11e40fcd31"} Nov 02 17:47:53.692193 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:53.692153 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" event={"ID":"134da1f2-4c24-495f-9b0a-76d21ba04df4","Type":"ContainerStarted","Data":"19c0b39c21858081f7ca89acdc091209edcfb1e0a6d204213cc8327e5726ba52"} Nov 02 17:47:53.692410 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:53.692203 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" Nov 02 17:47:53.739764 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:53.739705 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" podStartSLOduration=1.380985261 podStartE2EDuration="1.739689067s" totalImagesPullingTime="358.703806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:53.734838037 +0000 UTC m=+1092.356197815" watchObservedRunningTime="2026-11-02 17:47:53.739689067 +0000 UTC m=+1092.361048844" Nov 02 17:47:59.708326 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:47:59.708289 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" Nov 02 17:48:21.816976 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:48:21.816936 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:48:23.167143 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:23.164915 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z"] Nov 02 17:48:23.241078 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:23.238629 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27c0bb11-m5q9z"] Nov 02 17:48:23.501248 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:23.501164 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd" path="/var/lib/kubelet/pods/eacd9905-aa4b-4e8d-ba8c-bd16c6791bfd/volumes" Nov 02 17:48:31.148100 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:31.148060 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb"] Nov 02 17:48:31.158349 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:31.158322 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ec2929-w7skb"] Nov 02 17:48:31.500942 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:31.500819 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e97ccd-b641-413d-8aab-8f56f5c0e51d" path="/var/lib/kubelet/pods/21e97ccd-b641-413d-8aab-8f56f5c0e51d/volumes" Nov 02 17:48:36.737713 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.737671 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv"] Nov 02 17:48:36.742646 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.742617 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.769146 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.769115 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv"] Nov 02 17:48:36.804171 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.804143 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-504de7-openshift-default-dockercfg-f4lbh\"" Nov 02 17:48:36.839601 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.839571 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69187cd1-8708-46d3-91d9-62b9a9446454-istio-podinfo\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.839766 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.839620 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-credential-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.839766 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.839748 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-istio-data\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.839899 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.839776 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-istio-envoy\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.839899 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.839807 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-758gj\" (UniqueName: \"kubernetes.io/projected/69187cd1-8708-46d3-91d9-62b9a9446454-kube-api-access-758gj\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.839979 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.839963 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69187cd1-8708-46d3-91d9-62b9a9446454-istiod-ca-cert\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.840046 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.840031 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-workload-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.840159 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.840138 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69187cd1-8708-46d3-91d9-62b9a9446454-istio-token\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.840268 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.840252 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-workload-certs\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.941937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.941898 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69187cd1-8708-46d3-91d9-62b9a9446454-istio-token\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.942120 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.942008 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-workload-certs\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.942120 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.942082 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69187cd1-8708-46d3-91d9-62b9a9446454-istio-podinfo\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.942217 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.942129 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-credential-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.942217 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.942207 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-istio-data\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.942296 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.942234 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-istio-envoy\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.942296 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.942276 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-758gj\" (UniqueName: \"kubernetes.io/projected/69187cd1-8708-46d3-91d9-62b9a9446454-kube-api-access-758gj\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.942408 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.942378 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69187cd1-8708-46d3-91d9-62b9a9446454-istiod-ca-cert\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.942489 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.942470 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-workload-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.942553 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.942505 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-credential-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.942650 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.942618 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-istio-data\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.942757 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.942733 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-workload-certs\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.942920 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.942828 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-workload-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.943273 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.943247 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69187cd1-8708-46d3-91d9-62b9a9446454-istiod-ca-cert\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.944446 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.944420 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-istio-envoy\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.944724 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.944708 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69187cd1-8708-46d3-91d9-62b9a9446454-istio-podinfo\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.965590 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.965567 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69187cd1-8708-46d3-91d9-62b9a9446454-istio-token\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:36.966099 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:36.966079 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-758gj\" (UniqueName: \"kubernetes.io/projected/69187cd1-8708-46d3-91d9-62b9a9446454-kube-api-access-758gj\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:37.060525 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:37.060493 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:37.396813 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:48:37.396781 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69187cd1_8708_46d3_91d9_62b9a9446454.slice/crio-686c0617db29ec51d21261a0f1e8bdbf0471e3ac36ae6e543e4a2fe89015c1d4 WatchSource:0}: Error finding container 686c0617db29ec51d21261a0f1e8bdbf0471e3ac36ae6e543e4a2fe89015c1d4: Status 404 returned error can't find the container with id 686c0617db29ec51d21261a0f1e8bdbf0471e3ac36ae6e543e4a2fe89015c1d4 Nov 02 17:48:37.398209 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:37.398185 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv"] Nov 02 17:48:37.399549 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:37.399515 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:48:37.399639 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:37.399583 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:48:37.399639 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:37.399610 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:48:38.207938 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:38.207294 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" event={"ID":"69187cd1-8708-46d3-91d9-62b9a9446454","Type":"ContainerStarted","Data":"96e00330e0c4510c2cbc659ef7d91aa95e1de4096577fe0fa4fbd6592a0f3461"} Nov 02 17:48:38.207938 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:38.207396 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" event={"ID":"69187cd1-8708-46d3-91d9-62b9a9446454","Type":"ContainerStarted","Data":"686c0617db29ec51d21261a0f1e8bdbf0471e3ac36ae6e543e4a2fe89015c1d4"} Nov 02 17:48:39.062874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:39.062413 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:39.062874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:39.062523 4860 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 17:48:39.062874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:39.062577 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" podUID="69187cd1-8708-46d3-91d9-62b9a9446454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 17:48:40.071873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:40.069076 4860 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 17:48:40.071873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:40.069154 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" podUID="69187cd1-8708-46d3-91d9-62b9a9446454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 17:48:41.078916 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:41.071266 4860 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 17:48:41.078916 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:41.071338 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" podUID="69187cd1-8708-46d3-91d9-62b9a9446454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 17:48:41.835163 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:41.835113 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" podStartSLOduration=5.835098842 podStartE2EDuration="5.835098842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:38.290598601 +0000 UTC m=+1136.911958377" watchObservedRunningTime="2026-11-02 17:48:41.835098842 +0000 UTC m=+1140.456458618" Nov 02 17:48:41.836319 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:41.836300 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj"] Nov 02 17:48:41.840913 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:41.840894 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" Nov 02 17:48:41.866552 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:41.866520 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-c810ca-serving-cert\"" Nov 02 17:48:41.877595 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:41.877565 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj"] Nov 02 17:48:41.940733 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:41.940700 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vljrt\" (UniqueName: \"kubernetes.io/projected/42544ef4-8181-4c6e-bbd8-337a63fc0412-kube-api-access-vljrt\") pod \"maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj\" (UID: \"42544ef4-8181-4c6e-bbd8-337a63fc0412\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" Nov 02 17:48:41.940953 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:41.940917 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42544ef4-8181-4c6e-bbd8-337a63fc0412-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj\" (UID: \"42544ef4-8181-4c6e-bbd8-337a63fc0412\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" Nov 02 17:48:41.941067 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:41.941044 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42544ef4-8181-4c6e-bbd8-337a63fc0412-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj\" (UID: \"42544ef4-8181-4c6e-bbd8-337a63fc0412\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" Nov 02 17:48:41.941242 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:41.941225 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42544ef4-8181-4c6e-bbd8-337a63fc0412-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj\" (UID: \"42544ef4-8181-4c6e-bbd8-337a63fc0412\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" Nov 02 17:48:42.045535 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:42.045488 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42544ef4-8181-4c6e-bbd8-337a63fc0412-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj\" (UID: \"42544ef4-8181-4c6e-bbd8-337a63fc0412\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" Nov 02 17:48:42.046018 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:42.045996 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42544ef4-8181-4c6e-bbd8-337a63fc0412-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj\" (UID: \"42544ef4-8181-4c6e-bbd8-337a63fc0412\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" Nov 02 17:48:42.046333 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:42.046310 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vljrt\" (UniqueName: \"kubernetes.io/projected/42544ef4-8181-4c6e-bbd8-337a63fc0412-kube-api-access-vljrt\") pod \"maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj\" (UID: \"42544ef4-8181-4c6e-bbd8-337a63fc0412\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" Nov 02 17:48:42.046472 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:42.046455 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42544ef4-8181-4c6e-bbd8-337a63fc0412-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj\" (UID: \"42544ef4-8181-4c6e-bbd8-337a63fc0412\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" Nov 02 17:48:42.046737 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:48:42.046720 4860 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-serving-cert: secret "maas-api-e2e-shared-b-w5-c810ca-serving-cert" not found Nov 02 17:48:42.046903 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:48:42.046891 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42544ef4-8181-4c6e-bbd8-337a63fc0412-maas-api-tls podName:42544ef4-8181-4c6e-bbd8-337a63fc0412 nodeName:}" failed. No retries permitted until 2026-11-02 17:48:42.546867644 +0000 UTC m=+1141.168227417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42544ef4-8181-4c6e-bbd8-337a63fc0412-maas-api-tls") pod "maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" (UID: "42544ef4-8181-4c6e-bbd8-337a63fc0412") : secret "maas-api-e2e-shared-b-w5-c810ca-serving-cert" not found Nov 02 17:48:42.047902 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:42.047859 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42544ef4-8181-4c6e-bbd8-337a63fc0412-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj\" (UID: \"42544ef4-8181-4c6e-bbd8-337a63fc0412\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" Nov 02 17:48:42.049862 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:42.049823 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42544ef4-8181-4c6e-bbd8-337a63fc0412-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj\" (UID: \"42544ef4-8181-4c6e-bbd8-337a63fc0412\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" Nov 02 17:48:42.080444 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:42.078223 4860 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 17:48:42.080444 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:42.078284 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" podUID="69187cd1-8708-46d3-91d9-62b9a9446454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 17:48:42.101563 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:42.101440 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vljrt\" (UniqueName: \"kubernetes.io/projected/42544ef4-8181-4c6e-bbd8-337a63fc0412-kube-api-access-vljrt\") pod \"maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj\" (UID: \"42544ef4-8181-4c6e-bbd8-337a63fc0412\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" Nov 02 17:48:42.570876 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:42.561237 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42544ef4-8181-4c6e-bbd8-337a63fc0412-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj\" (UID: \"42544ef4-8181-4c6e-bbd8-337a63fc0412\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" Nov 02 17:48:42.585483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:42.585449 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42544ef4-8181-4c6e-bbd8-337a63fc0412-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj\" (UID: \"42544ef4-8181-4c6e-bbd8-337a63fc0412\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" Nov 02 17:48:42.770875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:42.767213 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" Nov 02 17:48:43.064405 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:43.064372 4860 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 17:48:43.089415 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:43.064668 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" podUID="69187cd1-8708-46d3-91d9-62b9a9446454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 17:48:43.199459 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:48:43.199414 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42544ef4_8181_4c6e_bbd8_337a63fc0412.slice/crio-c8a42c4c649bf7741672ecbff7c5bd3627397d9502ce8458b204960531fdc335 WatchSource:0}: Error finding container c8a42c4c649bf7741672ecbff7c5bd3627397d9502ce8458b204960531fdc335: Status 404 returned error can't find the container with id c8a42c4c649bf7741672ecbff7c5bd3627397d9502ce8458b204960531fdc335 Nov 02 17:48:43.219322 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:43.218138 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj"] Nov 02 17:48:43.312820 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:43.312638 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" event={"ID":"42544ef4-8181-4c6e-bbd8-337a63fc0412","Type":"ContainerStarted","Data":"c8a42c4c649bf7741672ecbff7c5bd3627397d9502ce8458b204960531fdc335"} Nov 02 17:48:44.052384 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:44.052345 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm"] Nov 02 17:48:44.057658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:44.057631 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cbeb6cc-2nzxm"] Nov 02 17:48:44.067366 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:44.067336 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:44.067692 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:44.067666 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:44.068507 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:44.068488 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 17:48:44.323237 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:44.323201 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" event={"ID":"42544ef4-8181-4c6e-bbd8-337a63fc0412","Type":"ContainerStarted","Data":"4acfd9abf14babf6e4dfb3c1347b27b38ca74931e7d643864f6d8bb9526ccb60"} Nov 02 17:48:44.323703 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:44.323261 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" Nov 02 17:48:44.375597 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:44.375546 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" podStartSLOduration=3.039911985 podStartE2EDuration="3.375531816s" totalImagesPullingTime="335.619831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:44.373171055 +0000 UTC m=+1142.994530831" watchObservedRunningTime="2026-11-02 17:48:44.375531816 +0000 UTC m=+1142.996891594" Nov 02 17:48:45.504518 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:45.504482 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2783bfae-7a20-40d8-bc9b-52a3de2f97f4" path="/var/lib/kubelet/pods/2783bfae-7a20-40d8-bc9b-52a3de2f97f4/volumes" Nov 02 17:48:47.567588 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:47.567555 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv"] Nov 02 17:48:47.572595 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:47.572574 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" Nov 02 17:48:47.590560 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:47.590536 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-04e682fe-serving-cert\"" Nov 02 17:48:47.599749 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:47.599720 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv"] Nov 02 17:48:47.726346 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:47.726304 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d06d023-fa6a-4acd-8e4d-06f74837ba66-metrics-certs\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv\" (UID: \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" Nov 02 17:48:47.726545 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:47.726426 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvlvn\" (UniqueName: \"kubernetes.io/projected/7d06d023-fa6a-4acd-8e4d-06f74837ba66-kube-api-access-cvlvn\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv\" (UID: \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" Nov 02 17:48:47.726661 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:47.726641 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d06d023-fa6a-4acd-8e4d-06f74837ba66-maas-api-tls\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv\" (UID: \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" Nov 02 17:48:47.726723 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:47.726667 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d06d023-fa6a-4acd-8e4d-06f74837ba66-openshift-service-ca\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv\" (UID: \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" Nov 02 17:48:47.828741 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:47.828638 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvlvn\" (UniqueName: \"kubernetes.io/projected/7d06d023-fa6a-4acd-8e4d-06f74837ba66-kube-api-access-cvlvn\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv\" (UID: \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" Nov 02 17:48:47.829095 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:47.829068 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d06d023-fa6a-4acd-8e4d-06f74837ba66-maas-api-tls\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv\" (UID: \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" Nov 02 17:48:47.829187 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:47.829101 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d06d023-fa6a-4acd-8e4d-06f74837ba66-openshift-service-ca\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv\" (UID: \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" Nov 02 17:48:47.829250 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:47.829219 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d06d023-fa6a-4acd-8e4d-06f74837ba66-metrics-certs\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv\" (UID: \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" Nov 02 17:48:47.830062 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:47.830031 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d06d023-fa6a-4acd-8e4d-06f74837ba66-openshift-service-ca\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv\" (UID: \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" Nov 02 17:48:47.832474 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:47.832452 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d06d023-fa6a-4acd-8e4d-06f74837ba66-maas-api-tls\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv\" (UID: \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" Nov 02 17:48:47.836646 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:47.836620 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d06d023-fa6a-4acd-8e4d-06f74837ba66-metrics-certs\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv\" (UID: \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" Nov 02 17:48:47.853684 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:47.853650 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvlvn\" (UniqueName: \"kubernetes.io/projected/7d06d023-fa6a-4acd-8e4d-06f74837ba66-kube-api-access-cvlvn\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv\" (UID: \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" Nov 02 17:48:47.888562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:47.888529 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" Nov 02 17:48:48.207735 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:48:48.207692 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d06d023_fa6a_4acd_8e4d_06f74837ba66.slice/crio-3463c2b7fedb27c2375ec1c02d447cf399576dbb20068cd0453a0b6d75750a72 WatchSource:0}: Error finding container 3463c2b7fedb27c2375ec1c02d447cf399576dbb20068cd0453a0b6d75750a72: Status 404 returned error can't find the container with id 3463c2b7fedb27c2375ec1c02d447cf399576dbb20068cd0453a0b6d75750a72 Nov 02 17:48:48.208880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:48.208844 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv"] Nov 02 17:48:48.365462 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:48.365419 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" event={"ID":"7d06d023-fa6a-4acd-8e4d-06f74837ba66","Type":"ContainerStarted","Data":"3463c2b7fedb27c2375ec1c02d447cf399576dbb20068cd0453a0b6d75750a72"} Nov 02 17:48:49.379136 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:49.379083 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" event={"ID":"7d06d023-fa6a-4acd-8e4d-06f74837ba66","Type":"ContainerStarted","Data":"0cbf564e5e3e6e2421d8b62ea087ec3cffda5bb07feca8ce7770d76abae657ba"} Nov 02 17:48:49.379577 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:49.379188 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" Nov 02 17:48:49.424638 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:49.424583 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" podStartSLOduration=2.013525839 podStartE2EDuration="2.424568335s" totalImagesPullingTime="411.042496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:49.422160505 +0000 UTC m=+1148.043520281" watchObservedRunningTime="2026-11-02 17:48:49.424568335 +0000 UTC m=+1148.045928117" Nov 02 17:48:50.338490 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:50.338457 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" Nov 02 17:48:51.772075 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:48:51.772041 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:48:53.134871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:53.123291 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t"] Nov 02 17:48:53.159872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:53.149322 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" Nov 02 17:48:53.168874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:53.165834 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-c4dbe3-serving-cert\"" Nov 02 17:48:53.194872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:53.194059 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t"] Nov 02 17:48:53.231441 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:53.231402 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnzvz\" (UniqueName: \"kubernetes.io/projected/82dde012-8d96-4cc2-8887-99d8e9833f23-kube-api-access-cnzvz\") pod \"maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t\" (UID: \"82dde012-8d96-4cc2-8887-99d8e9833f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" Nov 02 17:48:53.231675 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:53.231502 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82dde012-8d96-4cc2-8887-99d8e9833f23-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t\" (UID: \"82dde012-8d96-4cc2-8887-99d8e9833f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" Nov 02 17:48:53.231675 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:53.231631 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82dde012-8d96-4cc2-8887-99d8e9833f23-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t\" (UID: \"82dde012-8d96-4cc2-8887-99d8e9833f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" Nov 02 17:48:53.231798 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:53.231711 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82dde012-8d96-4cc2-8887-99d8e9833f23-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t\" (UID: \"82dde012-8d96-4cc2-8887-99d8e9833f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" Nov 02 17:48:53.334393 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:53.334342 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82dde012-8d96-4cc2-8887-99d8e9833f23-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t\" (UID: \"82dde012-8d96-4cc2-8887-99d8e9833f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" Nov 02 17:48:53.334571 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:53.334547 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82dde012-8d96-4cc2-8887-99d8e9833f23-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t\" (UID: \"82dde012-8d96-4cc2-8887-99d8e9833f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" Nov 02 17:48:53.334668 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:53.334652 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82dde012-8d96-4cc2-8887-99d8e9833f23-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t\" (UID: \"82dde012-8d96-4cc2-8887-99d8e9833f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" Nov 02 17:48:53.334834 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:53.334818 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnzvz\" (UniqueName: \"kubernetes.io/projected/82dde012-8d96-4cc2-8887-99d8e9833f23-kube-api-access-cnzvz\") pod \"maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t\" (UID: \"82dde012-8d96-4cc2-8887-99d8e9833f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" Nov 02 17:48:53.338873 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:48:53.338527 4860 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-serving-cert: secret "maas-api-e2e-shared-b-w0-c4dbe3-serving-cert" not found Nov 02 17:48:53.338873 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:48:53.338600 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82dde012-8d96-4cc2-8887-99d8e9833f23-maas-api-tls podName:82dde012-8d96-4cc2-8887-99d8e9833f23 nodeName:}" failed. No retries permitted until 2026-11-02 17:48:53.838581987 +0000 UTC m=+1152.459941744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82dde012-8d96-4cc2-8887-99d8e9833f23-maas-api-tls") pod "maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" (UID: "82dde012-8d96-4cc2-8887-99d8e9833f23") : secret "maas-api-e2e-shared-b-w0-c4dbe3-serving-cert" not found Nov 02 17:48:53.355217 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:53.353949 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82dde012-8d96-4cc2-8887-99d8e9833f23-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t\" (UID: \"82dde012-8d96-4cc2-8887-99d8e9833f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" Nov 02 17:48:53.360008 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:53.359981 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnzvz\" (UniqueName: \"kubernetes.io/projected/82dde012-8d96-4cc2-8887-99d8e9833f23-kube-api-access-cnzvz\") pod \"maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t\" (UID: \"82dde012-8d96-4cc2-8887-99d8e9833f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" Nov 02 17:48:53.361901 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:53.360395 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82dde012-8d96-4cc2-8887-99d8e9833f23-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t\" (UID: \"82dde012-8d96-4cc2-8887-99d8e9833f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" Nov 02 17:48:53.843617 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:53.843574 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82dde012-8d96-4cc2-8887-99d8e9833f23-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t\" (UID: \"82dde012-8d96-4cc2-8887-99d8e9833f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" Nov 02 17:48:53.846276 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:53.846243 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82dde012-8d96-4cc2-8887-99d8e9833f23-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t\" (UID: \"82dde012-8d96-4cc2-8887-99d8e9833f23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" Nov 02 17:48:54.100451 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:54.100359 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" Nov 02 17:48:54.528775 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:54.524389 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t"] Nov 02 17:48:54.531877 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:48:54.531609 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82dde012_8d96_4cc2_8887_99d8e9833f23.slice/crio-08b33c11cd846e9cf0eebfc989c8552093c2abdda37caa252073e1d594500dc1 WatchSource:0}: Error finding container 08b33c11cd846e9cf0eebfc989c8552093c2abdda37caa252073e1d594500dc1: Status 404 returned error can't find the container with id 08b33c11cd846e9cf0eebfc989c8552093c2abdda37caa252073e1d594500dc1 Nov 02 17:48:55.395953 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:55.395923 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" Nov 02 17:48:55.538430 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:55.538391 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" event={"ID":"82dde012-8d96-4cc2-8887-99d8e9833f23","Type":"ContainerStarted","Data":"065ceb15eaee3960b17ebbbbe4a1aad30b044da2c95423818ecee1f3299983fb"} Nov 02 17:48:55.538962 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:55.538448 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" event={"ID":"82dde012-8d96-4cc2-8887-99d8e9833f23","Type":"ContainerStarted","Data":"08b33c11cd846e9cf0eebfc989c8552093c2abdda37caa252073e1d594500dc1"} Nov 02 17:48:55.538962 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:55.538472 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" Nov 02 17:48:55.588607 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:55.588551 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" podStartSLOduration=2.279247337 podStartE2EDuration="2.588533708s" totalImagesPullingTime="309.286371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:55.586770265 +0000 UTC m=+1154.208130038" watchObservedRunningTime="2026-11-02 17:48:55.588533708 +0000 UTC m=+1154.209893483" Nov 02 17:48:56.061355 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:56.061320 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7"] Nov 02 17:48:56.066617 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:56.066588 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cc5ad5e-hdbw7"] Nov 02 17:48:57.513683 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:48:57.513646 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54fb1262-f113-462f-a0ed-fba26c773f2e" path="/var/lib/kubelet/pods/54fb1262-f113-462f-a0ed-fba26c773f2e/volumes" Nov 02 17:49:01.566171 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:01.566134 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" Nov 02 17:49:17.087693 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:17.087653 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95"] Nov 02 17:49:17.088157 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:17.087913 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" podUID="b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" containerName="istio-proxy" containerID="cri-o://5666cf0e22c229c8e8f59e13b44f11436e7c48f43a2cb4e490afff0172dd7364" gracePeriod=30 Nov 02 17:49:21.779472 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:49:21.779428 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:49:22.395309 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.395262 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:49:22.588809 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.588645 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-workload-certs\") pod \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " Nov 02 17:49:22.589028 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.588824 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-podinfo\") pod \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " Nov 02 17:49:22.589028 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.588873 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-token\" (UniqueName: \"kubernetes.io/projected/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-token\") pod \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " Nov 02 17:49:22.589028 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.588906 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-data\") pod \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " Nov 02 17:49:22.589028 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.589011 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-kube-api-access-rz5vd\" (UniqueName: \"kubernetes.io/projected/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-kube-api-access-rz5vd\") pod \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " Nov 02 17:49:22.589283 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.589057 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-credential-socket\") pod \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " Nov 02 17:49:22.589283 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.589186 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-workload-certs" pod "b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" (UID: "b3078bbb-3ccf-46f7-9a76-bc9ff80995f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:22.589283 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.589202 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istiod-ca-cert\") pod \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " Nov 02 17:49:22.589570 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.589323 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-workload-socket\") pod \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " Nov 02 17:49:22.589570 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.589382 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-envoy\") pod \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\" (UID: \"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2\") " Nov 02 17:49:22.589907 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.589841 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istiod-ca-cert" pod "b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" (UID: "b3078bbb-3ccf-46f7-9a76-bc9ff80995f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:49:22.590239 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.590210 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:49:22.590239 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.590240 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:49:22.590561 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.590527 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-credential-socket" pod "b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" (UID: "b3078bbb-3ccf-46f7-9a76-bc9ff80995f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:22.590776 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.590755 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-data" pod "b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" (UID: "b3078bbb-3ccf-46f7-9a76-bc9ff80995f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:22.591200 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.590961 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-workload-socket" pod "b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" (UID: "b3078bbb-3ccf-46f7-9a76-bc9ff80995f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:22.592630 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.592605 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-envoy" pod "b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" (UID: "b3078bbb-3ccf-46f7-9a76-bc9ff80995f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:22.593119 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.593095 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-kube-api-access-rz5vd" pod "b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" (UID: "b3078bbb-3ccf-46f7-9a76-bc9ff80995f2"). InnerVolumeSpecName "kube-api-access-rz5vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:22.594723 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.594606 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-podinfo" pod "b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" (UID: "b3078bbb-3ccf-46f7-9a76-bc9ff80995f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:49:22.595608 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.595585 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-token" pod "b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" (UID: "b3078bbb-3ccf-46f7-9a76-bc9ff80995f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:22.693366 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.693278 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:49:22.693366 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.693310 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:49:22.693366 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.693337 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:49:22.693366 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.693350 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:49:22.693366 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.693363 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:49:22.693637 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.693376 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz5vd\" (UniqueName: \"kubernetes.io/projected/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-kube-api-access-rz5vd\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:49:22.693637 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.693389 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:49:22.917790 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.917756 4860 generic.go:356] "Generic (PLEG): container finished" podID="b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" containerID="5666cf0e22c229c8e8f59e13b44f11436e7c48f43a2cb4e490afff0172dd7364" exitCode=0 Nov 02 17:49:22.918309 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.917817 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" event={"ID":"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2","Type":"ContainerDied","Data":"5666cf0e22c229c8e8f59e13b44f11436e7c48f43a2cb4e490afff0172dd7364"} Nov 02 17:49:22.918309 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.917827 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" Nov 02 17:49:22.918309 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.917873 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95" event={"ID":"b3078bbb-3ccf-46f7-9a76-bc9ff80995f2","Type":"ContainerDied","Data":"1fb6f0722c35f3411570a76087a4f100e85df162fd9393c748e0d2b6ab5c1eb7"} Nov 02 17:49:22.918309 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.917901 4860 scope.go:122] "RemoveContainer" containerID="5666cf0e22c229c8e8f59e13b44f11436e7c48f43a2cb4e490afff0172dd7364" Nov 02 17:49:22.931492 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.931471 4860 scope.go:122] "RemoveContainer" containerID="5666cf0e22c229c8e8f59e13b44f11436e7c48f43a2cb4e490afff0172dd7364" Nov 02 17:49:22.931811 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:49:22.931790 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5666cf0e22c229c8e8f59e13b44f11436e7c48f43a2cb4e490afff0172dd7364\": container with ID starting with 5666cf0e22c229c8e8f59e13b44f11436e7c48f43a2cb4e490afff0172dd7364 not found: ID does not exist" containerID="5666cf0e22c229c8e8f59e13b44f11436e7c48f43a2cb4e490afff0172dd7364" Nov 02 17:49:22.931911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.931826 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5666cf0e22c229c8e8f59e13b44f11436e7c48f43a2cb4e490afff0172dd7364"} err="failed to get container status \"5666cf0e22c229c8e8f59e13b44f11436e7c48f43a2cb4e490afff0172dd7364\": rpc error: code = NotFound desc = could not find container \"5666cf0e22c229c8e8f59e13b44f11436e7c48f43a2cb4e490afff0172dd7364\": container with ID starting with 5666cf0e22c229c8e8f59e13b44f11436e7c48f43a2cb4e490afff0172dd7364 not found: ID does not exist" Nov 02 17:49:22.988105 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.988028 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95"] Nov 02 17:49:22.999568 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:22.999524 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-7sg95"] Nov 02 17:49:23.502261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:23.502227 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" path="/var/lib/kubelet/pods/b3078bbb-3ccf-46f7-9a76-bc9ff80995f2/volumes" Nov 02 17:49:24.052755 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:24.052719 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v"] Nov 02 17:49:24.063866 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:24.063820 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7eba10-lj86v"] Nov 02 17:49:25.054096 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.054061 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7"] Nov 02 17:49:25.067004 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.066973 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3dc45e8-t4cz7"] Nov 02 17:49:25.501419 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.501388 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80578a9c-ef49-49de-a95c-eb8d9e1a79bd" path="/var/lib/kubelet/pods/80578a9c-ef49-49de-a95c-eb8d9e1a79bd/volumes" Nov 02 17:49:25.501958 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.501939 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd9777f9-d834-4e07-8493-6bd7f392be05" path="/var/lib/kubelet/pods/cd9777f9-d834-4e07-8493-6bd7f392be05/volumes" Nov 02 17:49:25.750101 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.750066 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm"] Nov 02 17:49:25.750900 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.750882 4860 cpu_manager.go:414] "containerMap: removing container" podUID="b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" containerName="istio-proxy" Nov 02 17:49:25.750900 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.750900 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" containerName="istio-proxy" Nov 02 17:49:25.751064 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.751048 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3078bbb-3ccf-46f7-9a76-bc9ff80995f2" containerName="istio-proxy" Nov 02 17:49:25.756517 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.756462 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:25.766424 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.766400 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm"] Nov 02 17:49:25.779205 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.779177 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-04e682fe\"/\"openshift-service-ca.crt\"" Nov 02 17:49:25.791635 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.791599 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-04e682fe\"/\"e2e-lifecycle-model-04e682fe-kserve-self-signed-certs\"" Nov 02 17:49:25.791635 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.791607 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-04e682fe\"/\"kube-root-ca.crt\"" Nov 02 17:49:25.791860 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.791688 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-04e682fe\"/\"default-dockercfg-47lzk\"" Nov 02 17:49:25.847922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.847837 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-model-cache\") pod \"e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:25.848132 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.847946 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d8921d-5111-4a20-b19c-23cc4cd063fe-tls-certs\") pod \"e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:25.848132 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.848026 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-home\") pod \"e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:25.848132 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.848052 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-dshm\") pod \"e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:25.848317 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.848188 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-tmp-dir\") pod \"e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:25.848317 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.848259 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pplcs\" (UniqueName: \"kubernetes.io/projected/b9d8921d-5111-4a20-b19c-23cc4cd063fe-kube-api-access-pplcs\") pod \"e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:25.950919 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.950865 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-dshm\") pod \"e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:25.951139 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.951014 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-tmp-dir\") pod \"e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:25.951139 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.951111 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pplcs\" (UniqueName: \"kubernetes.io/projected/b9d8921d-5111-4a20-b19c-23cc4cd063fe-kube-api-access-pplcs\") pod \"e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:25.951315 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.951295 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-model-cache\") pod \"e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:25.951378 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.951327 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d8921d-5111-4a20-b19c-23cc4cd063fe-tls-certs\") pod \"e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:25.951439 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.951386 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-home\") pod \"e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:25.951439 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.951426 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-tmp-dir\") pod \"e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:25.951608 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.951587 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-model-cache\") pod \"e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:25.951764 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.951743 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-home\") pod \"e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:25.953810 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.953783 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-dshm\") pod \"e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:25.954129 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.954108 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d8921d-5111-4a20-b19c-23cc4cd063fe-tls-certs\") pod \"e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:25.984038 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:25.984009 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pplcs\" (UniqueName: \"kubernetes.io/projected/b9d8921d-5111-4a20-b19c-23cc4cd063fe-kube-api-access-pplcs\") pod \"e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:26.074913 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:26.072587 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:26.473604 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:26.473573 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm"] Nov 02 17:49:26.473778 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:49:26.473752 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9d8921d_5111_4a20_b19c_23cc4cd063fe.slice/crio-01a4f88cc878dada4dc9cb0681f5f58038e9164355f8d7acac85ed4d9f435e88 WatchSource:0}: Error finding container 01a4f88cc878dada4dc9cb0681f5f58038e9164355f8d7acac85ed4d9f435e88: Status 404 returned error can't find the container with id 01a4f88cc878dada4dc9cb0681f5f58038e9164355f8d7acac85ed4d9f435e88 Nov 02 17:49:26.970023 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:26.969983 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" event={"ID":"b9d8921d-5111-4a20-b19c-23cc4cd063fe","Type":"ContainerStarted","Data":"8e8f8787befe95609a462b146273c4376c926085821d9165c3dfc249a318fa3f"} Nov 02 17:49:26.970226 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:26.970039 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" event={"ID":"b9d8921d-5111-4a20-b19c-23cc4cd063fe","Type":"ContainerStarted","Data":"01a4f88cc878dada4dc9cb0681f5f58038e9164355f8d7acac85ed4d9f435e88"} Nov 02 17:49:27.037666 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:27.037607 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" podStartSLOduration=2.037593654 podStartE2EDuration="2.037593654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:27.032972736 +0000 UTC m=+1185.654332513" watchObservedRunningTime="2026-11-02 17:49:27.037593654 +0000 UTC m=+1185.658953430" Nov 02 17:49:27.073919 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:27.073885 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:28.064022 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:28.063982 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g"] Nov 02 17:49:28.085144 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:28.085106 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-093569f7-n2c8g"] Nov 02 17:49:29.501575 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:29.501535 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64d9245-f8e9-48b9-a762-7c6c187a7ae3" path="/var/lib/kubelet/pods/b64d9245-f8e9-48b9-a762-7c6c187a7ae3/volumes" Nov 02 17:49:30.945531 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:30.945496 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb"] Nov 02 17:49:30.950541 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:30.950514 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" Nov 02 17:49:30.974354 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:30.974292 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-f59572-serving-cert\"" Nov 02 17:49:30.978938 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:30.978887 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb"] Nov 02 17:49:30.981287 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:30.981117 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b9f322-2bf8-415b-a26a-3723a34c2482-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb\" (UID: \"71b9f322-2bf8-415b-a26a-3723a34c2482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" Nov 02 17:49:30.981423 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:30.981287 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlqlb\" (UniqueName: \"kubernetes.io/projected/71b9f322-2bf8-415b-a26a-3723a34c2482-kube-api-access-rlqlb\") pod \"maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb\" (UID: \"71b9f322-2bf8-415b-a26a-3723a34c2482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" Nov 02 17:49:30.981423 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:30.981358 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b9f322-2bf8-415b-a26a-3723a34c2482-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb\" (UID: \"71b9f322-2bf8-415b-a26a-3723a34c2482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" Nov 02 17:49:30.981524 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:30.981423 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b9f322-2bf8-415b-a26a-3723a34c2482-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb\" (UID: \"71b9f322-2bf8-415b-a26a-3723a34c2482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" Nov 02 17:49:31.083229 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:31.083197 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b9f322-2bf8-415b-a26a-3723a34c2482-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb\" (UID: \"71b9f322-2bf8-415b-a26a-3723a34c2482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" Nov 02 17:49:31.083391 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:31.083355 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlqlb\" (UniqueName: \"kubernetes.io/projected/71b9f322-2bf8-415b-a26a-3723a34c2482-kube-api-access-rlqlb\") pod \"maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb\" (UID: \"71b9f322-2bf8-415b-a26a-3723a34c2482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" Nov 02 17:49:31.083444 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:31.083392 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b9f322-2bf8-415b-a26a-3723a34c2482-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb\" (UID: \"71b9f322-2bf8-415b-a26a-3723a34c2482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" Nov 02 17:49:31.083444 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:31.083430 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b9f322-2bf8-415b-a26a-3723a34c2482-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb\" (UID: \"71b9f322-2bf8-415b-a26a-3723a34c2482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" Nov 02 17:49:31.083533 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:49:31.083508 4860 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-serving-cert: secret "maas-api-e2e-shared-a-w4-f59572-serving-cert" not found Nov 02 17:49:31.083603 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:49:31.083590 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b9f322-2bf8-415b-a26a-3723a34c2482-maas-api-tls podName:71b9f322-2bf8-415b-a26a-3723a34c2482 nodeName:}" failed. No retries permitted until 2026-11-02 17:49:31.583567322 +0000 UTC m=+1190.204927079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71b9f322-2bf8-415b-a26a-3723a34c2482-maas-api-tls") pod "maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" (UID: "71b9f322-2bf8-415b-a26a-3723a34c2482") : secret "maas-api-e2e-shared-a-w4-f59572-serving-cert" not found Nov 02 17:49:31.084139 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:31.084116 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b9f322-2bf8-415b-a26a-3723a34c2482-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb\" (UID: \"71b9f322-2bf8-415b-a26a-3723a34c2482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" Nov 02 17:49:31.086155 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:31.086124 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b9f322-2bf8-415b-a26a-3723a34c2482-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb\" (UID: \"71b9f322-2bf8-415b-a26a-3723a34c2482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" Nov 02 17:49:31.107555 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:31.107528 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlqlb\" (UniqueName: \"kubernetes.io/projected/71b9f322-2bf8-415b-a26a-3723a34c2482-kube-api-access-rlqlb\") pod \"maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb\" (UID: \"71b9f322-2bf8-415b-a26a-3723a34c2482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" Nov 02 17:49:31.591575 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:31.591532 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b9f322-2bf8-415b-a26a-3723a34c2482-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb\" (UID: \"71b9f322-2bf8-415b-a26a-3723a34c2482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" Nov 02 17:49:31.594074 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:31.594042 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b9f322-2bf8-415b-a26a-3723a34c2482-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb\" (UID: \"71b9f322-2bf8-415b-a26a-3723a34c2482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" Nov 02 17:49:31.875140 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:31.875047 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" Nov 02 17:49:32.356285 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:49:32.356245 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b9f322_2bf8_415b_a26a_3723a34c2482.slice/crio-c672c49e06684b4e09687352cbcc36668ccdf0ecc8bec9c1db529643a51af1de WatchSource:0}: Error finding container c672c49e06684b4e09687352cbcc36668ccdf0ecc8bec9c1db529643a51af1de: Status 404 returned error can't find the container with id c672c49e06684b4e09687352cbcc36668ccdf0ecc8bec9c1db529643a51af1de Nov 02 17:49:32.358270 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:32.358226 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb"] Nov 02 17:49:33.064733 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:33.064661 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" event={"ID":"71b9f322-2bf8-415b-a26a-3723a34c2482","Type":"ContainerStarted","Data":"0e4313dc71db38e631122f67a415f8ebf8c51292e3d58787ad7e65a4441a1c77"} Nov 02 17:49:33.065634 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:33.065587 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" event={"ID":"71b9f322-2bf8-415b-a26a-3723a34c2482","Type":"ContainerStarted","Data":"c672c49e06684b4e09687352cbcc36668ccdf0ecc8bec9c1db529643a51af1de"} Nov 02 17:49:33.068075 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:33.067490 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" Nov 02 17:49:33.114943 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:33.114871 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" podStartSLOduration=2.749176996 podStartE2EDuration="3.114835152s" totalImagesPullingTime="365.658156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:33.112652869 +0000 UTC m=+1191.734012646" watchObservedRunningTime="2026-11-02 17:49:33.114835152 +0000 UTC m=+1191.736194929" Nov 02 17:49:34.051221 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:34.051188 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp"] Nov 02 17:49:34.063901 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:34.063868 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13938481-tlknp"] Nov 02 17:49:35.501794 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:35.501761 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="894756e3-8380-48c0-8881-0bfa433893f0" path="/var/lib/kubelet/pods/894756e3-8380-48c0-8881-0bfa433893f0/volumes" Nov 02 17:49:36.073212 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:36.073178 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:36.086178 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:36.086145 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:36.107454 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:36.107428 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:49:39.082314 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:39.082284 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" Nov 02 17:49:41.732584 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:41.732555 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 17:49:41.738526 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:41.738502 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 17:49:41.746426 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:41.746399 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 17:49:41.751766 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:41.751731 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 17:49:42.110873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:42.096737 4860 scope.go:122] "RemoveContainer" containerID="27c6064c268abe285052971d82a9a9950cce9ef3d0227d6ebb857069771832f5" Nov 02 17:49:42.222056 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:42.222025 4860 scope.go:122] "RemoveContainer" containerID="14fcd77ca9fa079a506abe2f3c7d6294b73a17709096784f978fc03710415cc1" Nov 02 17:49:42.248715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:42.248569 4860 scope.go:122] "RemoveContainer" containerID="b43ff5722336e269ebea67c8918a2ee1c7bd95a56918a9f3bbc0abd56a668fc1" Nov 02 17:49:51.770763 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:49:51.770729 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:49:52.053759 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:52.053681 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x"] Nov 02 17:49:52.065764 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:52.065735 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5c20415-bwb8x"] Nov 02 17:49:53.501536 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:53.501454 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ebc34e1-d76e-42e6-8f14-a29d41a93f5c" path="/var/lib/kubelet/pods/0ebc34e1-d76e-42e6-8f14-a29d41a93f5c/volumes" Nov 02 17:49:56.044527 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:56.044494 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs"] Nov 02 17:49:56.056382 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:56.056357 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a49143b-j87xs"] Nov 02 17:49:57.501698 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:57.501659 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20e25244-8eaa-42e5-8b9f-3324fee7a288" path="/var/lib/kubelet/pods/20e25244-8eaa-42e5-8b9f-3324fee7a288/volumes" Nov 02 17:49:57.803388 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:57.803314 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb"] Nov 02 17:49:57.810413 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:57.810387 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" Nov 02 17:49:57.826206 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:57.826180 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f99b8b-serving-cert\"" Nov 02 17:49:57.842606 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:57.842582 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb"] Nov 02 17:49:57.912214 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:57.912173 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-openshift-service-ca\") pod \"maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb\" (UID: \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" Nov 02 17:49:57.912214 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:57.912216 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-maas-api-tls\") pod \"maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb\" (UID: \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" Nov 02 17:49:57.912413 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:57.912301 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-metrics-certs\") pod \"maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb\" (UID: \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" Nov 02 17:49:57.912413 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:57.912355 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-448ln\" (UniqueName: \"kubernetes.io/projected/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-kube-api-access-448ln\") pod \"maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb\" (UID: \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" Nov 02 17:49:58.019211 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:58.019162 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-openshift-service-ca\") pod \"maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb\" (UID: \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" Nov 02 17:49:58.024885 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:58.019444 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-maas-api-tls\") pod \"maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb\" (UID: \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" Nov 02 17:49:58.024885 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:58.022505 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-metrics-certs\") pod \"maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb\" (UID: \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" Nov 02 17:49:58.024885 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:58.022602 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-448ln\" (UniqueName: \"kubernetes.io/projected/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-kube-api-access-448ln\") pod \"maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb\" (UID: \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" Nov 02 17:49:58.024885 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:58.021182 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-openshift-service-ca\") pod \"maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb\" (UID: \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" Nov 02 17:49:58.028888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:58.028841 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-metrics-certs\") pod \"maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb\" (UID: \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" Nov 02 17:49:58.066828 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:58.066794 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-maas-api-tls\") pod \"maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb\" (UID: \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" Nov 02 17:49:58.067082 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:58.066794 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-448ln\" (UniqueName: \"kubernetes.io/projected/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-kube-api-access-448ln\") pod \"maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb\" (UID: \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" Nov 02 17:49:58.137733 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:58.137700 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" Nov 02 17:49:58.462764 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:49:58.462733 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e468c0c_6c07_4aaf_b6b9_b5add41ee48a.slice/crio-33ad724c616bd37a235b4f7a8876f50ad4c7d74206dc51ad62e7c8fd23a4e498 WatchSource:0}: Error finding container 33ad724c616bd37a235b4f7a8876f50ad4c7d74206dc51ad62e7c8fd23a4e498: Status 404 returned error can't find the container with id 33ad724c616bd37a235b4f7a8876f50ad4c7d74206dc51ad62e7c8fd23a4e498 Nov 02 17:49:58.463034 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:58.463009 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb"] Nov 02 17:49:58.553375 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:58.553337 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" event={"ID":"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a","Type":"ContainerStarted","Data":"33ad724c616bd37a235b4f7a8876f50ad4c7d74206dc51ad62e7c8fd23a4e498"} Nov 02 17:49:59.566990 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:59.566949 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" event={"ID":"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a","Type":"ContainerStarted","Data":"40e7c44992a1818cb1fa7f2f38153190184b8921deb366542673b3c040a71440"} Nov 02 17:49:59.567420 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:59.567019 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" Nov 02 17:49:59.617179 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:49:59.617110 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" podStartSLOduration=2.181050857 podStartE2EDuration="2.61709643s" totalImagesPullingTime="436.045573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:59.612324213 +0000 UTC m=+1218.233683989" watchObservedRunningTime="2026-11-02 17:49:59.61709643 +0000 UTC m=+1218.238456240" Nov 02 17:50:05.581261 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:05.581224 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" Nov 02 17:50:19.078426 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:19.078389 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l"] Nov 02 17:50:19.094077 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:19.094044 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba7f293-d6w4l"] Nov 02 17:50:19.501359 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:19.501284 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70a3c054-80b7-4a3c-890d-7e38e1ba30cb" path="/var/lib/kubelet/pods/70a3c054-80b7-4a3c-890d-7e38e1ba30cb/volumes" Nov 02 17:50:21.434930 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:21.434896 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms"] Nov 02 17:50:21.439511 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:21.439488 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" Nov 02 17:50:21.457174 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:21.457149 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-786c42-serving-cert\"" Nov 02 17:50:21.490866 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:21.490820 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms"] Nov 02 17:50:21.545165 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:21.545127 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03e60aa0-7765-4a84-b221-a93829d5707a-maas-api-tls\") pod \"maas-api-e2e-api-a-786c42-7dc59f755f-8mlms\" (UID: \"03e60aa0-7765-4a84-b221-a93829d5707a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" Nov 02 17:50:21.545165 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:21.545163 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e60aa0-7765-4a84-b221-a93829d5707a-openshift-service-ca\") pod \"maas-api-e2e-api-a-786c42-7dc59f755f-8mlms\" (UID: \"03e60aa0-7765-4a84-b221-a93829d5707a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" Nov 02 17:50:21.545432 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:21.545398 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22rqj\" (UniqueName: \"kubernetes.io/projected/03e60aa0-7765-4a84-b221-a93829d5707a-kube-api-access-22rqj\") pod \"maas-api-e2e-api-a-786c42-7dc59f755f-8mlms\" (UID: \"03e60aa0-7765-4a84-b221-a93829d5707a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" Nov 02 17:50:21.545562 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:21.545543 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e60aa0-7765-4a84-b221-a93829d5707a-metrics-certs\") pod \"maas-api-e2e-api-a-786c42-7dc59f755f-8mlms\" (UID: \"03e60aa0-7765-4a84-b221-a93829d5707a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" Nov 02 17:50:21.647404 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:21.647370 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03e60aa0-7765-4a84-b221-a93829d5707a-maas-api-tls\") pod \"maas-api-e2e-api-a-786c42-7dc59f755f-8mlms\" (UID: \"03e60aa0-7765-4a84-b221-a93829d5707a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" Nov 02 17:50:21.647404 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:21.647406 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e60aa0-7765-4a84-b221-a93829d5707a-openshift-service-ca\") pod \"maas-api-e2e-api-a-786c42-7dc59f755f-8mlms\" (UID: \"03e60aa0-7765-4a84-b221-a93829d5707a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" Nov 02 17:50:21.647621 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:21.647465 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22rqj\" (UniqueName: \"kubernetes.io/projected/03e60aa0-7765-4a84-b221-a93829d5707a-kube-api-access-22rqj\") pod \"maas-api-e2e-api-a-786c42-7dc59f755f-8mlms\" (UID: \"03e60aa0-7765-4a84-b221-a93829d5707a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" Nov 02 17:50:21.647621 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:21.647606 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e60aa0-7765-4a84-b221-a93829d5707a-metrics-certs\") pod \"maas-api-e2e-api-a-786c42-7dc59f755f-8mlms\" (UID: \"03e60aa0-7765-4a84-b221-a93829d5707a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" Nov 02 17:50:21.648259 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:21.648238 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e60aa0-7765-4a84-b221-a93829d5707a-openshift-service-ca\") pod \"maas-api-e2e-api-a-786c42-7dc59f755f-8mlms\" (UID: \"03e60aa0-7765-4a84-b221-a93829d5707a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" Nov 02 17:50:21.649992 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:21.649970 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e60aa0-7765-4a84-b221-a93829d5707a-metrics-certs\") pod \"maas-api-e2e-api-a-786c42-7dc59f755f-8mlms\" (UID: \"03e60aa0-7765-4a84-b221-a93829d5707a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" Nov 02 17:50:21.650090 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:21.650016 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03e60aa0-7765-4a84-b221-a93829d5707a-maas-api-tls\") pod \"maas-api-e2e-api-a-786c42-7dc59f755f-8mlms\" (UID: \"03e60aa0-7765-4a84-b221-a93829d5707a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" Nov 02 17:50:21.674957 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:21.673281 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22rqj\" (UniqueName: \"kubernetes.io/projected/03e60aa0-7765-4a84-b221-a93829d5707a-kube-api-access-22rqj\") pod \"maas-api-e2e-api-a-786c42-7dc59f755f-8mlms\" (UID: \"03e60aa0-7765-4a84-b221-a93829d5707a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" Nov 02 17:50:21.756476 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:21.756389 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" Nov 02 17:50:21.785697 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:50:21.782827 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:50:22.094315 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:22.094282 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms"] Nov 02 17:50:22.094457 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:50:22.094361 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e60aa0_7765_4a84_b221_a93829d5707a.slice/crio-ac7cfe4175111f626f17f018869136a3f008b80ccb71952727b3bc292f65729f WatchSource:0}: Error finding container ac7cfe4175111f626f17f018869136a3f008b80ccb71952727b3bc292f65729f: Status 404 returned error can't find the container with id ac7cfe4175111f626f17f018869136a3f008b80ccb71952727b3bc292f65729f Nov 02 17:50:22.955769 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:22.955732 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" event={"ID":"03e60aa0-7765-4a84-b221-a93829d5707a","Type":"ContainerStarted","Data":"556795d15ef441eccd32244ab24006e506b6e5921a69d99dd698dedb6f191263"} Nov 02 17:50:22.956205 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:22.955793 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" event={"ID":"03e60aa0-7765-4a84-b221-a93829d5707a","Type":"ContainerStarted","Data":"ac7cfe4175111f626f17f018869136a3f008b80ccb71952727b3bc292f65729f"} Nov 02 17:50:22.956205 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:22.955821 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" Nov 02 17:50:23.003800 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:23.003749 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" podStartSLOduration=1.690257465 podStartE2EDuration="2.003733622s" totalImagesPullingTime="313.476157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:23.000994388 +0000 UTC m=+1241.622354165" watchObservedRunningTime="2026-11-02 17:50:23.003733622 +0000 UTC m=+1241.625093411" Nov 02 17:50:28.971122 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:28.971088 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" Nov 02 17:50:34.040387 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:34.040346 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6"] Nov 02 17:50:34.049282 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:34.049253 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b76de4cb-x4vf6"] Nov 02 17:50:35.501648 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:35.501609 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8" path="/var/lib/kubelet/pods/bcd80fd3-2e25-45cb-9d0c-7ccfdb99eda8/volumes" Nov 02 17:50:42.367630 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:42.367594 4860 scope.go:122] "RemoveContainer" containerID="0c232439398b5394e2e5c093aa26dea009123a0cab7dde5c3459aefce6b5b3a8" Nov 02 17:50:42.379466 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:42.379443 4860 scope.go:122] "RemoveContainer" containerID="a8156fb909b3efbf0c813157814c2f36614fc35287910d0a8e599bfe041e0567" Nov 02 17:50:42.390013 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:42.389987 4860 scope.go:122] "RemoveContainer" containerID="2c7b548f1b2202ec1a2407168d404f489a1d27f09f52e29011a04f91f1fae80c" Nov 02 17:50:42.400209 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:42.400185 4860 scope.go:122] "RemoveContainer" containerID="d27fe7c9af124f4dd4a0506947da14a1f09edbca0f95deed1b72cf6506732e20" Nov 02 17:50:42.411095 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:50:42.411071 4860 scope.go:122] "RemoveContainer" containerID="9bea48887da5b717cdd5bb7c00ccc6901b66552e3c6d8e5efbcdd38d27f5086f" Nov 02 17:50:51.775228 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:50:51.775182 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:51:02.533788 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:02.533751 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz"] Nov 02 17:51:02.534349 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:02.533982 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" podUID="0102a8d6-f853-4e95-aed3-3e033d85e970" containerName="istio-proxy" containerID="cri-o://d5f65253ddf3706b952f384952c36e618fba85411689c0b99a87558698ace0ed" gracePeriod=30 Nov 02 17:51:07.843483 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.843458 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:51:07.892166 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.892130 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-workload-socket\") pod \"0102a8d6-f853-4e95-aed3-3e033d85e970\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " Nov 02 17:51:07.892326 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.892293 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0102a8d6-f853-4e95-aed3-3e033d85e970-kube-api-access-x5pd4\" (UniqueName: \"kubernetes.io/projected/0102a8d6-f853-4e95-aed3-3e033d85e970-kube-api-access-x5pd4\") pod \"0102a8d6-f853-4e95-aed3-3e033d85e970\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " Nov 02 17:51:07.892421 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.892405 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-workload-certs\") pod \"0102a8d6-f853-4e95-aed3-3e033d85e970\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " Nov 02 17:51:07.892461 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.892439 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-data\") pod \"0102a8d6-f853-4e95-aed3-3e033d85e970\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " Nov 02 17:51:07.892461 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.892440 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-workload-socket" pod "0102a8d6-f853-4e95-aed3-3e033d85e970" (UID: "0102a8d6-f853-4e95-aed3-3e033d85e970"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:07.892560 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.892495 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0102a8d6-f853-4e95-aed3-3e033d85e970-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0102a8d6-f853-4e95-aed3-3e033d85e970-istiod-ca-cert\") pod \"0102a8d6-f853-4e95-aed3-3e033d85e970\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " Nov 02 17:51:07.892560 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.892555 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-credential-socket\") pod \"0102a8d6-f853-4e95-aed3-3e033d85e970\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " Nov 02 17:51:07.892666 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.892643 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-token\" (UniqueName: \"kubernetes.io/projected/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-token\") pod \"0102a8d6-f853-4e95-aed3-3e033d85e970\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " Nov 02 17:51:07.892722 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.892674 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-podinfo\") pod \"0102a8d6-f853-4e95-aed3-3e033d85e970\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " Nov 02 17:51:07.892775 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.892763 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-envoy\") pod \"0102a8d6-f853-4e95-aed3-3e033d85e970\" (UID: \"0102a8d6-f853-4e95-aed3-3e033d85e970\") " Nov 02 17:51:07.892844 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.892645 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-workload-certs" pod "0102a8d6-f853-4e95-aed3-3e033d85e970" (UID: "0102a8d6-f853-4e95-aed3-3e033d85e970"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:07.892979 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.892898 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-credential-socket" pod "0102a8d6-f853-4e95-aed3-3e033d85e970" (UID: "0102a8d6-f853-4e95-aed3-3e033d85e970"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:07.892979 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.892914 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-data" pod "0102a8d6-f853-4e95-aed3-3e033d85e970" (UID: "0102a8d6-f853-4e95-aed3-3e033d85e970"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:07.893333 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.893305 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0102a8d6-f853-4e95-aed3-3e033d85e970-istiod-ca-cert" pod "0102a8d6-f853-4e95-aed3-3e033d85e970" (UID: "0102a8d6-f853-4e95-aed3-3e033d85e970"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:07.893796 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.893703 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:51:07.893796 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.893729 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:51:07.893796 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.893746 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:51:07.893796 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.893760 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0102a8d6-f853-4e95-aed3-3e033d85e970-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:51:07.893796 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.893774 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:51:07.894728 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.894708 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0102a8d6-f853-4e95-aed3-3e033d85e970-kube-api-access-x5pd4" pod "0102a8d6-f853-4e95-aed3-3e033d85e970" (UID: "0102a8d6-f853-4e95-aed3-3e033d85e970"). InnerVolumeSpecName "kube-api-access-x5pd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:07.895203 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.895184 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-token" pod "0102a8d6-f853-4e95-aed3-3e033d85e970" (UID: "0102a8d6-f853-4e95-aed3-3e033d85e970"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:07.895379 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.895360 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-podinfo" pod "0102a8d6-f853-4e95-aed3-3e033d85e970" (UID: "0102a8d6-f853-4e95-aed3-3e033d85e970"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:51:07.895426 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.895381 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-envoy" pod "0102a8d6-f853-4e95-aed3-3e033d85e970" (UID: "0102a8d6-f853-4e95-aed3-3e033d85e970"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:07.995918 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.995880 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5pd4\" (UniqueName: \"kubernetes.io/projected/0102a8d6-f853-4e95-aed3-3e033d85e970-kube-api-access-x5pd4\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:51:07.995918 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.995909 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:51:07.995918 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.995918 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:51:07.995918 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:07.995926 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0102a8d6-f853-4e95-aed3-3e033d85e970-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:51:08.482825 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:08.482790 4860 generic.go:356] "Generic (PLEG): container finished" podID="0102a8d6-f853-4e95-aed3-3e033d85e970" containerID="d5f65253ddf3706b952f384952c36e618fba85411689c0b99a87558698ace0ed" exitCode=0 Nov 02 17:51:08.483046 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:08.482839 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" Nov 02 17:51:08.483046 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:08.482918 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" event={"ID":"0102a8d6-f853-4e95-aed3-3e033d85e970","Type":"ContainerDied","Data":"d5f65253ddf3706b952f384952c36e618fba85411689c0b99a87558698ace0ed"} Nov 02 17:51:08.483046 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:08.482975 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz" event={"ID":"0102a8d6-f853-4e95-aed3-3e033d85e970","Type":"ContainerDied","Data":"7ae6fdfae31a4fe0f12ae4822b218f391fe611c0831aa9be8fa16488cc4585fc"} Nov 02 17:51:08.483046 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:08.482999 4860 scope.go:122] "RemoveContainer" containerID="d5f65253ddf3706b952f384952c36e618fba85411689c0b99a87558698ace0ed" Nov 02 17:51:08.495402 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:08.495383 4860 scope.go:122] "RemoveContainer" containerID="d5f65253ddf3706b952f384952c36e618fba85411689c0b99a87558698ace0ed" Nov 02 17:51:08.495667 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:51:08.495651 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f65253ddf3706b952f384952c36e618fba85411689c0b99a87558698ace0ed\": container with ID starting with d5f65253ddf3706b952f384952c36e618fba85411689c0b99a87558698ace0ed not found: ID does not exist" containerID="d5f65253ddf3706b952f384952c36e618fba85411689c0b99a87558698ace0ed" Nov 02 17:51:08.495710 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:08.495675 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f65253ddf3706b952f384952c36e618fba85411689c0b99a87558698ace0ed"} err="failed to get container status \"d5f65253ddf3706b952f384952c36e618fba85411689c0b99a87558698ace0ed\": rpc error: code = NotFound desc = could not find container \"d5f65253ddf3706b952f384952c36e618fba85411689c0b99a87558698ace0ed\": container with ID starting with d5f65253ddf3706b952f384952c36e618fba85411689c0b99a87558698ace0ed not found: ID does not exist" Nov 02 17:51:08.566373 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:08.566333 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz"] Nov 02 17:51:08.578721 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:08.578684 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-kmbjz"] Nov 02 17:51:09.502197 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:09.502158 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0102a8d6-f853-4e95-aed3-3e033d85e970" path="/var/lib/kubelet/pods/0102a8d6-f853-4e95-aed3-3e033d85e970/volumes" Nov 02 17:51:21.772988 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:51:21.772952 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:51:42.466191 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:42.466156 4860 scope.go:122] "RemoveContainer" containerID="c2b085a793fa7983e85ce9cfe1bb65091bb7602809ce1365545471593cc788c3" Nov 02 17:51:42.477325 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:42.477301 4860 scope.go:122] "RemoveContainer" containerID="4009e24fe1459c92af99d12e102e62d2f088033bd3d3e5c0a6c22db77a12d42f" Nov 02 17:51:42.487221 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:42.487197 4860 scope.go:122] "RemoveContainer" containerID="d0179fc051b3bb7797894b87db3c72ed0758bab6d09f8280fb01657b1656c58f" Nov 02 17:51:42.497188 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:51:42.497171 4860 scope.go:122] "RemoveContainer" containerID="915c8b85826241947ab75e3268ed6afaef0c443937ff431a12d5b76c77d58b0d" Nov 02 17:51:51.774087 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:51:51.774045 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:52:21.773773 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:52:21.773734 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:52:32.126561 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:32.126522 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv"] Nov 02 17:52:32.127114 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:32.126784 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" podUID="7d06d023-fa6a-4acd-8e4d-06f74837ba66" containerName="maas-api" containerID="cri-o://0cbf564e5e3e6e2421d8b62ea087ec3cffda5bb07feca8ce7770d76abae657ba" gracePeriod=30 Nov 02 17:52:32.562929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:32.561894 4860 generic.go:356] "Generic (PLEG): container finished" podID="7d06d023-fa6a-4acd-8e4d-06f74837ba66" containerID="0cbf564e5e3e6e2421d8b62ea087ec3cffda5bb07feca8ce7770d76abae657ba" exitCode=0 Nov 02 17:52:32.562929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:32.562030 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" event={"ID":"7d06d023-fa6a-4acd-8e4d-06f74837ba66","Type":"ContainerDied","Data":"0cbf564e5e3e6e2421d8b62ea087ec3cffda5bb07feca8ce7770d76abae657ba"} Nov 02 17:52:32.642213 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:32.642174 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" Nov 02 17:52:32.672717 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:32.672148 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d06d023-fa6a-4acd-8e4d-06f74837ba66-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d06d023-fa6a-4acd-8e4d-06f74837ba66-maas-api-tls\") pod \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\" (UID: \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\") " Nov 02 17:52:32.673574 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:32.673519 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d06d023-fa6a-4acd-8e4d-06f74837ba66-openshift-service-ca" pod "7d06d023-fa6a-4acd-8e4d-06f74837ba66" (UID: "7d06d023-fa6a-4acd-8e4d-06f74837ba66"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:32.673806 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:32.673009 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d06d023-fa6a-4acd-8e4d-06f74837ba66-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d06d023-fa6a-4acd-8e4d-06f74837ba66-openshift-service-ca\") pod \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\" (UID: \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\") " Nov 02 17:52:32.674182 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:32.674163 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d06d023-fa6a-4acd-8e4d-06f74837ba66-kube-api-access-cvlvn\" (UniqueName: \"kubernetes.io/projected/7d06d023-fa6a-4acd-8e4d-06f74837ba66-kube-api-access-cvlvn\") pod \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\" (UID: \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\") " Nov 02 17:52:32.675781 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:32.675760 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d06d023-fa6a-4acd-8e4d-06f74837ba66-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d06d023-fa6a-4acd-8e4d-06f74837ba66-metrics-certs\") pod \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\" (UID: \"7d06d023-fa6a-4acd-8e4d-06f74837ba66\") " Nov 02 17:52:32.677148 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:32.677129 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d06d023-fa6a-4acd-8e4d-06f74837ba66-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:32.715130 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:32.695960 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d06d023-fa6a-4acd-8e4d-06f74837ba66-metrics-certs" pod "7d06d023-fa6a-4acd-8e4d-06f74837ba66" (UID: "7d06d023-fa6a-4acd-8e4d-06f74837ba66"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:52:32.715130 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:32.696080 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d06d023-fa6a-4acd-8e4d-06f74837ba66-kube-api-access-cvlvn" pod "7d06d023-fa6a-4acd-8e4d-06f74837ba66" (UID: "7d06d023-fa6a-4acd-8e4d-06f74837ba66"). InnerVolumeSpecName "kube-api-access-cvlvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:32.715130 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:32.696231 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d06d023-fa6a-4acd-8e4d-06f74837ba66-maas-api-tls" pod "7d06d023-fa6a-4acd-8e4d-06f74837ba66" (UID: "7d06d023-fa6a-4acd-8e4d-06f74837ba66"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:52:32.783872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:32.783058 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvlvn\" (UniqueName: \"kubernetes.io/projected/7d06d023-fa6a-4acd-8e4d-06f74837ba66-kube-api-access-cvlvn\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:32.783872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:32.783094 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d06d023-fa6a-4acd-8e4d-06f74837ba66-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:32.783872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:32.783107 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d06d023-fa6a-4acd-8e4d-06f74837ba66-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:33.571835 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:33.571769 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" event={"ID":"7d06d023-fa6a-4acd-8e4d-06f74837ba66","Type":"ContainerDied","Data":"3463c2b7fedb27c2375ec1c02d447cf399576dbb20068cd0453a0b6d75750a72"} Nov 02 17:52:33.571835 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:33.571795 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv" Nov 02 17:52:33.572411 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:33.571883 4860 scope.go:122] "RemoveContainer" containerID="0cbf564e5e3e6e2421d8b62ea087ec3cffda5bb07feca8ce7770d76abae657ba" Nov 02 17:52:33.599810 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:52:33.599779 4860 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d06d023_fa6a_4acd_8e4d_06f74837ba66.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d06d023_fa6a_4acd_8e4d_06f74837ba66.slice/crio-3463c2b7fedb27c2375ec1c02d447cf399576dbb20068cd0453a0b6d75750a72\": RecentStats: unable to find data in memory cache]" Nov 02 17:52:33.627233 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:33.627204 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv"] Nov 02 17:52:33.638791 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:33.638761 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-9cjqv"] Nov 02 17:52:35.502795 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:35.502763 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d06d023-fa6a-4acd-8e4d-06f74837ba66" path="/var/lib/kubelet/pods/7d06d023-fa6a-4acd-8e4d-06f74837ba66/volumes" Nov 02 17:52:42.553198 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:42.553160 4860 scope.go:122] "RemoveContainer" containerID="2ae7f11da07eb9ebb3678b90453f5700b253932fbc0f682ac98088cd8146ddc9" Nov 02 17:52:47.916986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:47.916951 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9"] Nov 02 17:52:47.917410 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:47.917174 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" containerID="cri-o://761ba773b36c43ac72ea1f952ef31299046ae47d74a186f0e80f6c5a5c9220dd" gracePeriod=30 Nov 02 17:52:48.654178 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:48.654129 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l"] Nov 02 17:52:48.654420 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:48.654373 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" podUID="64e985f1-f451-4ee1-ba69-76ce8949ca40" containerName="istio-proxy" containerID="cri-o://9ab0311433b9c038b398b73f30b5f6c7f2ca8f3a5a69d1fe911e59a954a72c97" gracePeriod=30 Nov 02 17:52:50.009417 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:50.009382 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:52:50.009814 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:50.009442 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:52:51.776020 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:52:51.775984 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:52:53.213487 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.213456 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:52:53.254638 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.254607 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-istio-envoy\") pod \"8568f684-1f8d-444f-854a-5e54b0484861\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " Nov 02 17:52:53.254825 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.254653 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8568f684-1f8d-444f-854a-5e54b0484861-kube-api-access-qdjg6\" (UniqueName: \"kubernetes.io/projected/8568f684-1f8d-444f-854a-5e54b0484861-kube-api-access-qdjg6\") pod \"8568f684-1f8d-444f-854a-5e54b0484861\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " Nov 02 17:52:53.254825 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.254714 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8568f684-1f8d-444f-854a-5e54b0484861-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8568f684-1f8d-444f-854a-5e54b0484861-istiod-ca-cert\") pod \"8568f684-1f8d-444f-854a-5e54b0484861\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " Nov 02 17:52:53.254825 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.254745 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-credential-socket\") pod \"8568f684-1f8d-444f-854a-5e54b0484861\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " Nov 02 17:52:53.255046 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.254895 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8568f684-1f8d-444f-854a-5e54b0484861-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8568f684-1f8d-444f-854a-5e54b0484861-istio-podinfo\") pod \"8568f684-1f8d-444f-854a-5e54b0484861\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " Nov 02 17:52:53.255046 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.254942 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-workload-socket\") pod \"8568f684-1f8d-444f-854a-5e54b0484861\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " Nov 02 17:52:53.255046 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.254981 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-workload-certs\") pod \"8568f684-1f8d-444f-854a-5e54b0484861\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " Nov 02 17:52:53.255211 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.255131 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8568f684-1f8d-444f-854a-5e54b0484861-istio-token\" (UniqueName: \"kubernetes.io/projected/8568f684-1f8d-444f-854a-5e54b0484861-istio-token\") pod \"8568f684-1f8d-444f-854a-5e54b0484861\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " Nov 02 17:52:53.255211 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.255143 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-credential-socket" pod "8568f684-1f8d-444f-854a-5e54b0484861" (UID: "8568f684-1f8d-444f-854a-5e54b0484861"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:53.255211 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.255205 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-istio-data\") pod \"8568f684-1f8d-444f-854a-5e54b0484861\" (UID: \"8568f684-1f8d-444f-854a-5e54b0484861\") " Nov 02 17:52:53.255361 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.255230 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8568f684-1f8d-444f-854a-5e54b0484861-istiod-ca-cert" pod "8568f684-1f8d-444f-854a-5e54b0484861" (UID: "8568f684-1f8d-444f-854a-5e54b0484861"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:53.255497 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.255467 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-workload-socket" pod "8568f684-1f8d-444f-854a-5e54b0484861" (UID: "8568f684-1f8d-444f-854a-5e54b0484861"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:53.255643 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.255552 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-workload-certs" pod "8568f684-1f8d-444f-854a-5e54b0484861" (UID: "8568f684-1f8d-444f-854a-5e54b0484861"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:53.255723 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.255672 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-istio-data" pod "8568f684-1f8d-444f-854a-5e54b0484861" (UID: "8568f684-1f8d-444f-854a-5e54b0484861"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:53.256026 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.256007 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:53.256026 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.256025 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8568f684-1f8d-444f-854a-5e54b0484861-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:53.256171 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.256034 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:53.256171 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.256043 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:53.256171 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.256051 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:53.257258 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.257229 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8568f684-1f8d-444f-854a-5e54b0484861-istio-token" pod "8568f684-1f8d-444f-854a-5e54b0484861" (UID: "8568f684-1f8d-444f-854a-5e54b0484861"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:53.257258 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.257246 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8568f684-1f8d-444f-854a-5e54b0484861-kube-api-access-qdjg6" pod "8568f684-1f8d-444f-854a-5e54b0484861" (UID: "8568f684-1f8d-444f-854a-5e54b0484861"). InnerVolumeSpecName "kube-api-access-qdjg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:53.257406 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.257387 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-istio-envoy" pod "8568f684-1f8d-444f-854a-5e54b0484861" (UID: "8568f684-1f8d-444f-854a-5e54b0484861"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:53.257466 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.257429 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8568f684-1f8d-444f-854a-5e54b0484861-istio-podinfo" pod "8568f684-1f8d-444f-854a-5e54b0484861" (UID: "8568f684-1f8d-444f-854a-5e54b0484861"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:52:53.357668 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.357633 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8568f684-1f8d-444f-854a-5e54b0484861-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:53.357668 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.357663 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8568f684-1f8d-444f-854a-5e54b0484861-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:53.357668 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.357674 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdjg6\" (UniqueName: \"kubernetes.io/projected/8568f684-1f8d-444f-854a-5e54b0484861-kube-api-access-qdjg6\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:53.357932 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.357686 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8568f684-1f8d-444f-854a-5e54b0484861-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:53.757866 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.757802 4860 generic.go:356] "Generic (PLEG): container finished" podID="8568f684-1f8d-444f-854a-5e54b0484861" containerID="761ba773b36c43ac72ea1f952ef31299046ae47d74a186f0e80f6c5a5c9220dd" exitCode=0 Nov 02 17:52:53.758057 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.757882 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" Nov 02 17:52:53.758057 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.757897 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" event={"ID":"8568f684-1f8d-444f-854a-5e54b0484861","Type":"ContainerDied","Data":"761ba773b36c43ac72ea1f952ef31299046ae47d74a186f0e80f6c5a5c9220dd"} Nov 02 17:52:53.758057 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.757973 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9" event={"ID":"8568f684-1f8d-444f-854a-5e54b0484861","Type":"ContainerDied","Data":"6d5d08cfe410a56630c12751d150eb423e66a6d752be0c6a83c46a39fb2e6f0b"} Nov 02 17:52:53.758057 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.758004 4860 scope.go:122] "RemoveContainer" containerID="761ba773b36c43ac72ea1f952ef31299046ae47d74a186f0e80f6c5a5c9220dd" Nov 02 17:52:53.770467 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.770450 4860 scope.go:122] "RemoveContainer" containerID="761ba773b36c43ac72ea1f952ef31299046ae47d74a186f0e80f6c5a5c9220dd" Nov 02 17:52:53.770719 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:52:53.770699 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"761ba773b36c43ac72ea1f952ef31299046ae47d74a186f0e80f6c5a5c9220dd\": container with ID starting with 761ba773b36c43ac72ea1f952ef31299046ae47d74a186f0e80f6c5a5c9220dd not found: ID does not exist" containerID="761ba773b36c43ac72ea1f952ef31299046ae47d74a186f0e80f6c5a5c9220dd" Nov 02 17:52:53.770777 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.770727 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761ba773b36c43ac72ea1f952ef31299046ae47d74a186f0e80f6c5a5c9220dd"} err="failed to get container status \"761ba773b36c43ac72ea1f952ef31299046ae47d74a186f0e80f6c5a5c9220dd\": rpc error: code = NotFound desc = could not find container \"761ba773b36c43ac72ea1f952ef31299046ae47d74a186f0e80f6c5a5c9220dd\": container with ID starting with 761ba773b36c43ac72ea1f952ef31299046ae47d74a186f0e80f6c5a5c9220dd not found: ID does not exist" Nov 02 17:52:53.814410 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.814378 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9"] Nov 02 17:52:53.827934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.827903 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-zkqm9"] Nov 02 17:52:53.985945 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:53.985918 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:52:54.075756 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.075724 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-podinfo\") pod \"64e985f1-f451-4ee1-ba69-76ce8949ca40\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " Nov 02 17:52:54.076019 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.075778 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e985f1-f451-4ee1-ba69-76ce8949ca40-kube-api-access-c7dwp\" (UniqueName: \"kubernetes.io/projected/64e985f1-f451-4ee1-ba69-76ce8949ca40-kube-api-access-c7dwp\") pod \"64e985f1-f451-4ee1-ba69-76ce8949ca40\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " Nov 02 17:52:54.076019 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.075816 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-workload-socket\") pod \"64e985f1-f451-4ee1-ba69-76ce8949ca40\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " Nov 02 17:52:54.076019 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.076000 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-credential-socket\") pod \"64e985f1-f451-4ee1-ba69-76ce8949ca40\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " Nov 02 17:52:54.076195 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.076037 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-data\") pod \"64e985f1-f451-4ee1-ba69-76ce8949ca40\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " Nov 02 17:52:54.076195 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.076171 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-token\" (UniqueName: \"kubernetes.io/projected/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-token\") pod \"64e985f1-f451-4ee1-ba69-76ce8949ca40\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " Nov 02 17:52:54.076195 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.076174 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-workload-socket" pod "64e985f1-f451-4ee1-ba69-76ce8949ca40" (UID: "64e985f1-f451-4ee1-ba69-76ce8949ca40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:54.076372 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.076356 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64e985f1-f451-4ee1-ba69-76ce8949ca40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e985f1-f451-4ee1-ba69-76ce8949ca40-istiod-ca-cert\") pod \"64e985f1-f451-4ee1-ba69-76ce8949ca40\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " Nov 02 17:52:54.076431 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.076363 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-credential-socket" pod "64e985f1-f451-4ee1-ba69-76ce8949ca40" (UID: "64e985f1-f451-4ee1-ba69-76ce8949ca40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:54.076431 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.076373 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-data" pod "64e985f1-f451-4ee1-ba69-76ce8949ca40" (UID: "64e985f1-f451-4ee1-ba69-76ce8949ca40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:54.076431 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.076399 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-envoy\") pod \"64e985f1-f451-4ee1-ba69-76ce8949ca40\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " Nov 02 17:52:54.076575 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.076436 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-workload-certs\") pod \"64e985f1-f451-4ee1-ba69-76ce8949ca40\" (UID: \"64e985f1-f451-4ee1-ba69-76ce8949ca40\") " Nov 02 17:52:54.076629 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.076608 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e985f1-f451-4ee1-ba69-76ce8949ca40-istiod-ca-cert" pod "64e985f1-f451-4ee1-ba69-76ce8949ca40" (UID: "64e985f1-f451-4ee1-ba69-76ce8949ca40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:54.077143 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.077117 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:54.077143 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.077140 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:54.077298 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.077154 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e985f1-f451-4ee1-ba69-76ce8949ca40-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:54.077298 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.077166 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:54.078683 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.078654 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-podinfo" pod "64e985f1-f451-4ee1-ba69-76ce8949ca40" (UID: "64e985f1-f451-4ee1-ba69-76ce8949ca40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:52:54.079114 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.079089 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e985f1-f451-4ee1-ba69-76ce8949ca40-kube-api-access-c7dwp" pod "64e985f1-f451-4ee1-ba69-76ce8949ca40" (UID: "64e985f1-f451-4ee1-ba69-76ce8949ca40"). InnerVolumeSpecName "kube-api-access-c7dwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:54.079415 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.079396 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-envoy" pod "64e985f1-f451-4ee1-ba69-76ce8949ca40" (UID: "64e985f1-f451-4ee1-ba69-76ce8949ca40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:54.080134 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.080109 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-workload-certs" pod "64e985f1-f451-4ee1-ba69-76ce8949ca40" (UID: "64e985f1-f451-4ee1-ba69-76ce8949ca40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:54.086065 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.086039 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-token" pod "64e985f1-f451-4ee1-ba69-76ce8949ca40" (UID: "64e985f1-f451-4ee1-ba69-76ce8949ca40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:54.179032 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.178994 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:54.179032 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.179040 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:54.179270 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.179055 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e985f1-f451-4ee1-ba69-76ce8949ca40-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:54.179270 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.179066 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e985f1-f451-4ee1-ba69-76ce8949ca40-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:54.179270 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.179079 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7dwp\" (UniqueName: \"kubernetes.io/projected/64e985f1-f451-4ee1-ba69-76ce8949ca40-kube-api-access-c7dwp\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:52:54.768591 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.768560 4860 generic.go:356] "Generic (PLEG): container finished" podID="64e985f1-f451-4ee1-ba69-76ce8949ca40" containerID="9ab0311433b9c038b398b73f30b5f6c7f2ca8f3a5a69d1fe911e59a954a72c97" exitCode=0 Nov 02 17:52:54.769148 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.768614 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" Nov 02 17:52:54.769148 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.768639 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" event={"ID":"64e985f1-f451-4ee1-ba69-76ce8949ca40","Type":"ContainerDied","Data":"9ab0311433b9c038b398b73f30b5f6c7f2ca8f3a5a69d1fe911e59a954a72c97"} Nov 02 17:52:54.769148 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.768690 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l" event={"ID":"64e985f1-f451-4ee1-ba69-76ce8949ca40","Type":"ContainerDied","Data":"045f6e4591981038557a2faf47c1ce35b57d1087429383a6a2f5a463d306fc8d"} Nov 02 17:52:54.769148 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.768716 4860 scope.go:122] "RemoveContainer" containerID="9ab0311433b9c038b398b73f30b5f6c7f2ca8f3a5a69d1fe911e59a954a72c97" Nov 02 17:52:54.782212 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.782193 4860 scope.go:122] "RemoveContainer" containerID="9ab0311433b9c038b398b73f30b5f6c7f2ca8f3a5a69d1fe911e59a954a72c97" Nov 02 17:52:54.782449 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:52:54.782431 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab0311433b9c038b398b73f30b5f6c7f2ca8f3a5a69d1fe911e59a954a72c97\": container with ID starting with 9ab0311433b9c038b398b73f30b5f6c7f2ca8f3a5a69d1fe911e59a954a72c97 not found: ID does not exist" containerID="9ab0311433b9c038b398b73f30b5f6c7f2ca8f3a5a69d1fe911e59a954a72c97" Nov 02 17:52:54.782510 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.782457 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab0311433b9c038b398b73f30b5f6c7f2ca8f3a5a69d1fe911e59a954a72c97"} err="failed to get container status \"9ab0311433b9c038b398b73f30b5f6c7f2ca8f3a5a69d1fe911e59a954a72c97\": rpc error: code = NotFound desc = could not find container \"9ab0311433b9c038b398b73f30b5f6c7f2ca8f3a5a69d1fe911e59a954a72c97\": container with ID starting with 9ab0311433b9c038b398b73f30b5f6c7f2ca8f3a5a69d1fe911e59a954a72c97 not found: ID does not exist" Nov 02 17:52:54.837301 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.837266 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l"] Nov 02 17:52:54.848446 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:54.848419 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dnj4l"] Nov 02 17:52:55.514785 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:55.514736 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64e985f1-f451-4ee1-ba69-76ce8949ca40" path="/var/lib/kubelet/pods/64e985f1-f451-4ee1-ba69-76ce8949ca40/volumes" Nov 02 17:52:55.516061 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:52:55.516002 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8568f684-1f8d-444f-854a-5e54b0484861" path="/var/lib/kubelet/pods/8568f684-1f8d-444f-854a-5e54b0484861/volumes" Nov 02 17:53:10.606107 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.606068 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq"] Nov 02 17:53:10.606827 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.606812 4860 cpu_manager.go:414] "containerMap: removing container" podUID="7d06d023-fa6a-4acd-8e4d-06f74837ba66" containerName="maas-api" Nov 02 17:53:10.606957 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.606828 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d06d023-fa6a-4acd-8e4d-06f74837ba66" containerName="maas-api" Nov 02 17:53:10.607026 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.606969 4860 cpu_manager.go:414] "containerMap: removing container" podUID="0102a8d6-f853-4e95-aed3-3e033d85e970" containerName="istio-proxy" Nov 02 17:53:10.607026 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.606975 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0102a8d6-f853-4e95-aed3-3e033d85e970" containerName="istio-proxy" Nov 02 17:53:10.607026 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.607017 4860 cpu_manager.go:414] "containerMap: removing container" podUID="64e985f1-f451-4ee1-ba69-76ce8949ca40" containerName="istio-proxy" Nov 02 17:53:10.607026 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.607024 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e985f1-f451-4ee1-ba69-76ce8949ca40" containerName="istio-proxy" Nov 02 17:53:10.607183 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.607123 4860 cpu_manager.go:414] "containerMap: removing container" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" Nov 02 17:53:10.607183 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.607164 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" Nov 02 17:53:10.607408 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.607395 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="8568f684-1f8d-444f-854a-5e54b0484861" containerName="istio-proxy" Nov 02 17:53:10.607445 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.607416 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d06d023-fa6a-4acd-8e4d-06f74837ba66" containerName="maas-api" Nov 02 17:53:10.607445 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.607426 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0102a8d6-f853-4e95-aed3-3e033d85e970" containerName="istio-proxy" Nov 02 17:53:10.607445 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.607432 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e985f1-f451-4ee1-ba69-76ce8949ca40" containerName="istio-proxy" Nov 02 17:53:10.612080 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.612055 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" Nov 02 17:53:10.632414 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.632378 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq"] Nov 02 17:53:10.651072 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.651039 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-04e682fe-serving-cert\"" Nov 02 17:53:10.780793 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.780755 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-openshift-service-ca\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq\" (UID: \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" Nov 02 17:53:10.780973 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.780957 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-metrics-certs\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq\" (UID: \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" Nov 02 17:53:10.781050 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.781018 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jffh\" (UniqueName: \"kubernetes.io/projected/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-kube-api-access-8jffh\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq\" (UID: \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" Nov 02 17:53:10.781235 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.781210 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-maas-api-tls\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq\" (UID: \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" Nov 02 17:53:10.883191 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.883105 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-openshift-service-ca\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq\" (UID: \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" Nov 02 17:53:10.883350 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.883289 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-metrics-certs\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq\" (UID: \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" Nov 02 17:53:10.883402 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.883371 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jffh\" (UniqueName: \"kubernetes.io/projected/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-kube-api-access-8jffh\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq\" (UID: \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" Nov 02 17:53:10.883512 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.883498 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-maas-api-tls\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq\" (UID: \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" Nov 02 17:53:10.885384 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.885345 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-openshift-service-ca\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq\" (UID: \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" Nov 02 17:53:10.887290 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.887258 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-maas-api-tls\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq\" (UID: \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" Nov 02 17:53:10.889900 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.889874 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-metrics-certs\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq\" (UID: \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" Nov 02 17:53:10.913065 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.913030 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jffh\" (UniqueName: \"kubernetes.io/projected/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-kube-api-access-8jffh\") pod \"maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq\" (UID: \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" Nov 02 17:53:10.973094 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:10.973061 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" Nov 02 17:53:11.314458 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:11.314424 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq"] Nov 02 17:53:11.315472 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:53:11.315447 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6214eb11_a46c_49d2_8d4c_dfb6b0fc79c6.slice/crio-d4581a173891de6115ac04ed6c7e7e1291e4744b50e54ab605cee6e30f6879d2 WatchSource:0}: Error finding container d4581a173891de6115ac04ed6c7e7e1291e4744b50e54ab605cee6e30f6879d2: Status 404 returned error can't find the container with id d4581a173891de6115ac04ed6c7e7e1291e4744b50e54ab605cee6e30f6879d2 Nov 02 17:53:11.316757 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:11.316739 4860 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:53:11.993873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:11.987364 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" event={"ID":"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6","Type":"ContainerStarted","Data":"77a0f6f12cf12754b1729cddfd287883fef7c95aef59564e5cef37f419ce9895"} Nov 02 17:53:11.993873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:11.987448 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" event={"ID":"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6","Type":"ContainerStarted","Data":"d4581a173891de6115ac04ed6c7e7e1291e4744b50e54ab605cee6e30f6879d2"} Nov 02 17:53:11.993873 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:11.988644 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" Nov 02 17:53:12.059926 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:12.046559 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" podStartSLOduration=1.608492864 podStartE2EDuration="2.046540347s" totalImagesPullingTime="438.047483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:12.037578767 +0000 UTC m=+1410.658938537" watchObservedRunningTime="2026-11-02 17:53:12.046540347 +0000 UTC m=+1410.667900124" Nov 02 17:53:14.499872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:14.498163 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb"] Nov 02 17:53:14.499872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:14.498898 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" podUID="4e468c0c-6c07-4aaf-b6b9-b5add41ee48a" containerName="maas-api" containerID="cri-o://40e7c44992a1818cb1fa7f2f38153190184b8921deb366542673b3c040a71440" gracePeriod=30 Nov 02 17:53:15.088322 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:15.088124 4860 generic.go:356] "Generic (PLEG): container finished" podID="4e468c0c-6c07-4aaf-b6b9-b5add41ee48a" containerID="40e7c44992a1818cb1fa7f2f38153190184b8921deb366542673b3c040a71440" exitCode=0 Nov 02 17:53:15.088322 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:15.088202 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" event={"ID":"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a","Type":"ContainerDied","Data":"40e7c44992a1818cb1fa7f2f38153190184b8921deb366542673b3c040a71440"} Nov 02 17:53:15.279953 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:15.279922 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" Nov 02 17:53:15.371471 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:15.371377 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-metrics-certs\") pod \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\" (UID: \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\") " Nov 02 17:53:15.372236 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:15.372211 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-maas-api-tls\") pod \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\" (UID: \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\") " Nov 02 17:53:15.372489 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:15.372468 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-kube-api-access-448ln\" (UniqueName: \"kubernetes.io/projected/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-kube-api-access-448ln\") pod \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\" (UID: \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\") " Nov 02 17:53:15.372694 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:15.372674 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-openshift-service-ca\") pod \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\" (UID: \"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a\") " Nov 02 17:53:15.373201 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:15.373168 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-openshift-service-ca" pod "4e468c0c-6c07-4aaf-b6b9-b5add41ee48a" (UID: "4e468c0c-6c07-4aaf-b6b9-b5add41ee48a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:15.373586 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:15.373562 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:15.375715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:15.375679 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-maas-api-tls" pod "4e468c0c-6c07-4aaf-b6b9-b5add41ee48a" (UID: "4e468c0c-6c07-4aaf-b6b9-b5add41ee48a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:15.377389 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:15.377354 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-kube-api-access-448ln" pod "4e468c0c-6c07-4aaf-b6b9-b5add41ee48a" (UID: "4e468c0c-6c07-4aaf-b6b9-b5add41ee48a"). InnerVolumeSpecName "kube-api-access-448ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:15.377608 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:15.377579 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-metrics-certs" pod "4e468c0c-6c07-4aaf-b6b9-b5add41ee48a" (UID: "4e468c0c-6c07-4aaf-b6b9-b5add41ee48a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:15.475610 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:15.475575 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:15.475610 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:15.475604 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-448ln\" (UniqueName: \"kubernetes.io/projected/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-kube-api-access-448ln\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:15.475610 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:15.475617 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:16.097931 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:16.097899 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" Nov 02 17:53:16.097931 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:16.097915 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb" event={"ID":"4e468c0c-6c07-4aaf-b6b9-b5add41ee48a","Type":"ContainerDied","Data":"33ad724c616bd37a235b4f7a8876f50ad4c7d74206dc51ad62e7c8fd23a4e498"} Nov 02 17:53:16.098462 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:16.097987 4860 scope.go:122] "RemoveContainer" containerID="40e7c44992a1818cb1fa7f2f38153190184b8921deb366542673b3c040a71440" Nov 02 17:53:16.158898 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:16.158859 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb"] Nov 02 17:53:16.169257 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:16.169225 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f99b8b-5fb856f7bf-8c2fb"] Nov 02 17:53:17.515798 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:17.515760 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e468c0c-6c07-4aaf-b6b9-b5add41ee48a" path="/var/lib/kubelet/pods/4e468c0c-6c07-4aaf-b6b9-b5add41ee48a/volumes" Nov 02 17:53:19.064340 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:19.064305 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" Nov 02 17:53:21.255505 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.255468 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv"] Nov 02 17:53:21.256636 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.256614 4860 cpu_manager.go:414] "containerMap: removing container" podUID="4e468c0c-6c07-4aaf-b6b9-b5add41ee48a" containerName="maas-api" Nov 02 17:53:21.256636 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.256635 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e468c0c-6c07-4aaf-b6b9-b5add41ee48a" containerName="maas-api" Nov 02 17:53:21.256793 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.256783 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e468c0c-6c07-4aaf-b6b9-b5add41ee48a" containerName="maas-api" Nov 02 17:53:21.262388 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.262366 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.293553 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.293509 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c3c112-openshift-default-dockercfg-6fgt6\"" Nov 02 17:53:21.301991 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.301963 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv"] Nov 02 17:53:21.400300 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.400268 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvdvq\" (UniqueName: \"kubernetes.io/projected/cbdf6416-8c0d-40c7-978b-30a30f640b51-kube-api-access-dvdvq\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.400474 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.400400 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.400474 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.400454 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.400553 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.400473 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbdf6416-8c0d-40c7-978b-30a30f640b51-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.400585 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.400550 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.400707 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.400685 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.400745 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.400722 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.400778 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.400752 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.400901 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.400884 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.501808 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.501777 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.501808 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.501815 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.502055 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.501840 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.502055 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.501927 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.502055 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.502029 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvdvq\" (UniqueName: \"kubernetes.io/projected/cbdf6416-8c0d-40c7-978b-30a30f640b51-kube-api-access-dvdvq\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.502189 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.502154 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.502249 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.502187 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.502302 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.502253 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.502302 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.502270 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.502302 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.502274 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbdf6416-8c0d-40c7-978b-30a30f640b51-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.502440 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.502325 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.502440 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.502360 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.502588 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.502562 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.502938 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.502915 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbdf6416-8c0d-40c7-978b-30a30f640b51-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.504295 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.504275 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.504653 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.504632 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.525587 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.525525 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvdvq\" (UniqueName: \"kubernetes.io/projected/cbdf6416-8c0d-40c7-978b-30a30f640b51-kube-api-access-dvdvq\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.525587 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.525556 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.583410 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.583376 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:21.772796 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:53:21.772744 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:53:21.916127 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:53:21.916093 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbdf6416_8c0d_40c7_978b_30a30f640b51.slice/crio-57dccc6818253903e74ccda61e53c9bc24ac2cf377114432b579f60d84dbb747 WatchSource:0}: Error finding container 57dccc6818253903e74ccda61e53c9bc24ac2cf377114432b579f60d84dbb747: Status 404 returned error can't find the container with id 57dccc6818253903e74ccda61e53c9bc24ac2cf377114432b579f60d84dbb747 Nov 02 17:53:21.916718 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.916696 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv"] Nov 02 17:53:21.919077 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.919044 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:21.919148 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.919108 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:21.919148 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:21.919138 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:22.175278 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:22.175191 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" event={"ID":"cbdf6416-8c0d-40c7-978b-30a30f640b51","Type":"ContainerStarted","Data":"f22bd39d41120607cd0eb36ec9d99788b405a599c76082d9bdb627bd27c4b5e5"} Nov 02 17:53:22.175278 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:22.175263 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" event={"ID":"cbdf6416-8c0d-40c7-978b-30a30f640b51","Type":"ContainerStarted","Data":"57dccc6818253903e74ccda61e53c9bc24ac2cf377114432b579f60d84dbb747"} Nov 02 17:53:22.228002 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:22.227939 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" podStartSLOduration=1.227919399 podStartE2EDuration="1.227919399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:22.223056289 +0000 UTC m=+1420.844416066" watchObservedRunningTime="2026-11-02 17:53:22.227919399 +0000 UTC m=+1420.849279176" Nov 02 17:53:22.583653 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:22.583615 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:23.590874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:23.587635 4860 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 17:53:23.590874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:23.587716 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" podUID="cbdf6416-8c0d-40c7-978b-30a30f640b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 17:53:24.585227 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:24.585186 4860 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 17:53:24.585486 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:24.585259 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" podUID="cbdf6416-8c0d-40c7-978b-30a30f640b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 17:53:25.585801 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:25.585273 4860 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 17:53:25.585801 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:25.585337 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" podUID="cbdf6416-8c0d-40c7-978b-30a30f640b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 17:53:26.584715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:26.584663 4860 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 17:53:26.584954 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:26.584729 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" podUID="cbdf6416-8c0d-40c7-978b-30a30f640b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 17:53:27.584219 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:27.584182 4860 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 17:53:27.584646 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:27.584237 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" podUID="cbdf6416-8c0d-40c7-978b-30a30f640b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 17:53:28.588814 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:28.588778 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:28.589340 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:28.589217 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:28.590044 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:28.590024 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:53:41.616415 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:41.616354 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms"] Nov 02 17:53:41.617293 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:41.617260 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" podUID="03e60aa0-7765-4a84-b221-a93829d5707a" containerName="maas-api" containerID="cri-o://556795d15ef441eccd32244ab24006e506b6e5921a69d99dd698dedb6f191263" gracePeriod=30 Nov 02 17:53:42.130170 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.130142 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" Nov 02 17:53:42.220523 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.220485 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03e60aa0-7765-4a84-b221-a93829d5707a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e60aa0-7765-4a84-b221-a93829d5707a-metrics-certs\") pod \"03e60aa0-7765-4a84-b221-a93829d5707a\" (UID: \"03e60aa0-7765-4a84-b221-a93829d5707a\") " Nov 02 17:53:42.220727 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.220651 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e60aa0-7765-4a84-b221-a93829d5707a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e60aa0-7765-4a84-b221-a93829d5707a-openshift-service-ca\") pod \"03e60aa0-7765-4a84-b221-a93829d5707a\" (UID: \"03e60aa0-7765-4a84-b221-a93829d5707a\") " Nov 02 17:53:42.220727 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.220680 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03e60aa0-7765-4a84-b221-a93829d5707a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03e60aa0-7765-4a84-b221-a93829d5707a-maas-api-tls\") pod \"03e60aa0-7765-4a84-b221-a93829d5707a\" (UID: \"03e60aa0-7765-4a84-b221-a93829d5707a\") " Nov 02 17:53:42.220901 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.220768 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e60aa0-7765-4a84-b221-a93829d5707a-kube-api-access-22rqj\" (UniqueName: \"kubernetes.io/projected/03e60aa0-7765-4a84-b221-a93829d5707a-kube-api-access-22rqj\") pod \"03e60aa0-7765-4a84-b221-a93829d5707a\" (UID: \"03e60aa0-7765-4a84-b221-a93829d5707a\") " Nov 02 17:53:42.221799 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.221771 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e60aa0-7765-4a84-b221-a93829d5707a-openshift-service-ca" pod "03e60aa0-7765-4a84-b221-a93829d5707a" (UID: "03e60aa0-7765-4a84-b221-a93829d5707a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:42.227663 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.227630 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e60aa0-7765-4a84-b221-a93829d5707a-kube-api-access-22rqj" pod "03e60aa0-7765-4a84-b221-a93829d5707a" (UID: "03e60aa0-7765-4a84-b221-a93829d5707a"). InnerVolumeSpecName "kube-api-access-22rqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:42.232896 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.232862 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e60aa0-7765-4a84-b221-a93829d5707a-maas-api-tls" pod "03e60aa0-7765-4a84-b221-a93829d5707a" (UID: "03e60aa0-7765-4a84-b221-a93829d5707a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:42.242223 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.242190 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e60aa0-7765-4a84-b221-a93829d5707a-metrics-certs" pod "03e60aa0-7765-4a84-b221-a93829d5707a" (UID: "03e60aa0-7765-4a84-b221-a93829d5707a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:42.323926 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.323809 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e60aa0-7765-4a84-b221-a93829d5707a-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:42.324207 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.324184 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e60aa0-7765-4a84-b221-a93829d5707a-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:42.324357 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.324334 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03e60aa0-7765-4a84-b221-a93829d5707a-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:42.324441 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.324361 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22rqj\" (UniqueName: \"kubernetes.io/projected/03e60aa0-7765-4a84-b221-a93829d5707a-kube-api-access-22rqj\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:42.462577 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.462473 4860 generic.go:356] "Generic (PLEG): container finished" podID="03e60aa0-7765-4a84-b221-a93829d5707a" containerID="556795d15ef441eccd32244ab24006e506b6e5921a69d99dd698dedb6f191263" exitCode=0 Nov 02 17:53:42.462577 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.462551 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" Nov 02 17:53:42.462577 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.462558 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" event={"ID":"03e60aa0-7765-4a84-b221-a93829d5707a","Type":"ContainerDied","Data":"556795d15ef441eccd32244ab24006e506b6e5921a69d99dd698dedb6f191263"} Nov 02 17:53:42.462910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.462633 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms" event={"ID":"03e60aa0-7765-4a84-b221-a93829d5707a","Type":"ContainerDied","Data":"ac7cfe4175111f626f17f018869136a3f008b80ccb71952727b3bc292f65729f"} Nov 02 17:53:42.462910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.462663 4860 scope.go:122] "RemoveContainer" containerID="556795d15ef441eccd32244ab24006e506b6e5921a69d99dd698dedb6f191263" Nov 02 17:53:42.489022 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.488319 4860 scope.go:122] "RemoveContainer" containerID="556795d15ef441eccd32244ab24006e506b6e5921a69d99dd698dedb6f191263" Nov 02 17:53:42.489022 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:53:42.488744 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556795d15ef441eccd32244ab24006e506b6e5921a69d99dd698dedb6f191263\": container with ID starting with 556795d15ef441eccd32244ab24006e506b6e5921a69d99dd698dedb6f191263 not found: ID does not exist" containerID="556795d15ef441eccd32244ab24006e506b6e5921a69d99dd698dedb6f191263" Nov 02 17:53:42.489022 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.488787 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556795d15ef441eccd32244ab24006e506b6e5921a69d99dd698dedb6f191263"} err="failed to get container status \"556795d15ef441eccd32244ab24006e506b6e5921a69d99dd698dedb6f191263\": rpc error: code = NotFound desc = could not find container \"556795d15ef441eccd32244ab24006e506b6e5921a69d99dd698dedb6f191263\": container with ID starting with 556795d15ef441eccd32244ab24006e506b6e5921a69d99dd698dedb6f191263 not found: ID does not exist" Nov 02 17:53:42.577976 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.577933 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms"] Nov 02 17:53:42.606877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:42.598618 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-786c42-7dc59f755f-8mlms"] Nov 02 17:53:43.505290 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:43.505249 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e60aa0-7765-4a84-b221-a93829d5707a" path="/var/lib/kubelet/pods/03e60aa0-7765-4a84-b221-a93829d5707a/volumes" Nov 02 17:53:46.305607 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.305572 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8"] Nov 02 17:53:46.306335 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.306316 4860 cpu_manager.go:414] "containerMap: removing container" podUID="03e60aa0-7765-4a84-b221-a93829d5707a" containerName="maas-api" Nov 02 17:53:46.306398 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.306337 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e60aa0-7765-4a84-b221-a93829d5707a" containerName="maas-api" Nov 02 17:53:46.306496 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.306487 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e60aa0-7765-4a84-b221-a93829d5707a" containerName="maas-api" Nov 02 17:53:46.311701 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.311684 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" Nov 02 17:53:46.331741 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.331710 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8"] Nov 02 17:53:46.344251 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.344220 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b913b2-serving-cert\"" Nov 02 17:53:46.345797 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.345773 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6e9573f-76d6-4d96-a0a7-393ee6f63617-metrics-certs\") pod \"maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8\" (UID: \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" Nov 02 17:53:46.346001 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.345820 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e9573f-76d6-4d96-a0a7-393ee6f63617-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8\" (UID: \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" Nov 02 17:53:46.346172 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.346145 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6e9573f-76d6-4d96-a0a7-393ee6f63617-maas-api-tls\") pod \"maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8\" (UID: \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" Nov 02 17:53:46.346372 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.346214 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8n5h\" (UniqueName: \"kubernetes.io/projected/b6e9573f-76d6-4d96-a0a7-393ee6f63617-kube-api-access-s8n5h\") pod \"maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8\" (UID: \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" Nov 02 17:53:46.447304 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.447266 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6e9573f-76d6-4d96-a0a7-393ee6f63617-maas-api-tls\") pod \"maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8\" (UID: \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" Nov 02 17:53:46.447474 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.447329 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8n5h\" (UniqueName: \"kubernetes.io/projected/b6e9573f-76d6-4d96-a0a7-393ee6f63617-kube-api-access-s8n5h\") pod \"maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8\" (UID: \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" Nov 02 17:53:46.447474 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:53:46.447421 4860 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-serving-cert: secret "maas-api-e2e-inf-b-b913b2-serving-cert" not found Nov 02 17:53:46.447550 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:53:46.447510 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e9573f-76d6-4d96-a0a7-393ee6f63617-maas-api-tls podName:b6e9573f-76d6-4d96-a0a7-393ee6f63617 nodeName:}" failed. No retries permitted until 2026-11-02 17:53:46.947488924 +0000 UTC m=+1445.568848692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b6e9573f-76d6-4d96-a0a7-393ee6f63617-maas-api-tls") pod "maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" (UID: "b6e9573f-76d6-4d96-a0a7-393ee6f63617") : secret "maas-api-e2e-inf-b-b913b2-serving-cert" not found Nov 02 17:53:46.447608 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.447545 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6e9573f-76d6-4d96-a0a7-393ee6f63617-metrics-certs\") pod \"maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8\" (UID: \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" Nov 02 17:53:46.447608 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.447598 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e9573f-76d6-4d96-a0a7-393ee6f63617-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8\" (UID: \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" Nov 02 17:53:46.448505 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.448478 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e9573f-76d6-4d96-a0a7-393ee6f63617-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8\" (UID: \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" Nov 02 17:53:46.450039 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.450015 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6e9573f-76d6-4d96-a0a7-393ee6f63617-metrics-certs\") pod \"maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8\" (UID: \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" Nov 02 17:53:46.475726 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.475699 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8n5h\" (UniqueName: \"kubernetes.io/projected/b6e9573f-76d6-4d96-a0a7-393ee6f63617-kube-api-access-s8n5h\") pod \"maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8\" (UID: \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" Nov 02 17:53:46.956262 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.956226 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6e9573f-76d6-4d96-a0a7-393ee6f63617-maas-api-tls\") pod \"maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8\" (UID: \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" Nov 02 17:53:46.958844 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:46.958817 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6e9573f-76d6-4d96-a0a7-393ee6f63617-maas-api-tls\") pod \"maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8\" (UID: \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" Nov 02 17:53:47.229783 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:47.227887 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" Nov 02 17:53:47.597722 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:47.582573 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8"] Nov 02 17:53:48.013834 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:48.013805 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc"] Nov 02 17:53:48.014092 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:48.014065 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" containerName="istio-proxy" containerID="cri-o://1b307520b0bd5961ea1b07b32a774511c62adcc9ae774110d5dcd3c3cb6ebc83" gracePeriod=30 Nov 02 17:53:48.041581 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:48.041548 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p"] Nov 02 17:53:48.042029 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:48.041997 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" podUID="c3ed150c-7f9b-4422-b675-301dcb0c6e3f" containerName="istio-proxy" containerID="cri-o://feaa572512091083c7abc3a83f224471f3642af2ec63ca1775b70364353790db" gracePeriod=30 Nov 02 17:53:48.097081 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:48.097053 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns"] Nov 02 17:53:48.097509 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:48.097477 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" podUID="4db49741-b199-403e-98f4-fbba98d95b84" containerName="istio-proxy" containerID="cri-o://9e8497a541e5cf41a39fcbd5046b00f03ae31d270e373f585bb8a04db3da9df5" gracePeriod=30 Nov 02 17:53:48.115336 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:48.115273 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p"] Nov 02 17:53:48.115550 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:48.115518 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerName="istio-proxy" containerID="cri-o://c282f97739a44629edc3ae9cd598928f8f13e61f1231f7a88b930130e13c5cc3" gracePeriod=30 Nov 02 17:53:48.530655 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:48.530617 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" event={"ID":"b6e9573f-76d6-4d96-a0a7-393ee6f63617","Type":"ContainerStarted","Data":"198fcb0cd815d01444d707f1a9b56f62f4c5144328b3822d59ca43dc1399fa86"} Nov 02 17:53:48.530893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:48.530679 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" event={"ID":"b6e9573f-76d6-4d96-a0a7-393ee6f63617","Type":"ContainerStarted","Data":"d04976470774be5102f4dcc4999ae1f228e1ec15f96ea4e43926bf0c03ba8600"} Nov 02 17:53:48.530893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:48.530705 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" Nov 02 17:53:48.584328 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:48.584253 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" podStartSLOduration=2.159489405 podStartE2EDuration="2.584232124s" totalImagesPullingTime="424.742719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:48.576352872 +0000 UTC m=+1447.197712650" watchObservedRunningTime="2026-11-02 17:53:48.584232124 +0000 UTC m=+1447.205591902" Nov 02 17:53:48.736534 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:48.736080 4860 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:53:48.736534 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:48.736178 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:53:49.388304 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:49.388264 4860 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:53:49.388484 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:49.388334 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:53:51.772732 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:53:51.772695 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:53:51.814900 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:51.814824 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p"] Nov 02 17:53:51.819890 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:51.819828 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p" Nov 02 17:53:51.839511 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:51.839478 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p"] Nov 02 17:53:51.871948 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:51.871911 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e2e941-5c6e-4a08-913c-e2e56bef424b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p\" (UID: \"e7e2e941-5c6e-4a08-913c-e2e56bef424b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p" Nov 02 17:53:51.872144 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:51.872037 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mnnr\" (UniqueName: \"kubernetes.io/projected/e7e2e941-5c6e-4a08-913c-e2e56bef424b-kube-api-access-7mnnr\") pod \"maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p\" (UID: \"e7e2e941-5c6e-4a08-913c-e2e56bef424b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p" Nov 02 17:53:51.872202 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:51.872155 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e2e941-5c6e-4a08-913c-e2e56bef424b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p\" (UID: \"e7e2e941-5c6e-4a08-913c-e2e56bef424b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p" Nov 02 17:53:51.902382 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:51.902358 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 17:53:51.902532 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:51.902385 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:53:51.974539 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:51.974502 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e2e941-5c6e-4a08-913c-e2e56bef424b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p\" (UID: \"e7e2e941-5c6e-4a08-913c-e2e56bef424b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p" Nov 02 17:53:51.974737 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:51.974645 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mnnr\" (UniqueName: \"kubernetes.io/projected/e7e2e941-5c6e-4a08-913c-e2e56bef424b-kube-api-access-7mnnr\") pod \"maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p\" (UID: \"e7e2e941-5c6e-4a08-913c-e2e56bef424b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p" Nov 02 17:53:51.974803 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:51.974760 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e2e941-5c6e-4a08-913c-e2e56bef424b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p\" (UID: \"e7e2e941-5c6e-4a08-913c-e2e56bef424b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p" Nov 02 17:53:51.975406 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:51.975377 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e2e941-5c6e-4a08-913c-e2e56bef424b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p\" (UID: \"e7e2e941-5c6e-4a08-913c-e2e56bef424b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p" Nov 02 17:53:51.977041 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:51.977013 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e2e941-5c6e-4a08-913c-e2e56bef424b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p\" (UID: \"e7e2e941-5c6e-4a08-913c-e2e56bef424b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p" Nov 02 17:53:51.998241 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:51.998205 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mnnr\" (UniqueName: \"kubernetes.io/projected/e7e2e941-5c6e-4a08-913c-e2e56bef424b-kube-api-access-7mnnr\") pod \"maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p\" (UID: \"e7e2e941-5c6e-4a08-913c-e2e56bef424b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p" Nov 02 17:53:52.141165 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:52.141133 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p" Nov 02 17:53:52.457553 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:53:52.457475 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e2e941_5c6e_4a08_913c_e2e56bef424b.slice/crio-f085acc3b661800d9d594f01b15dd01f67f35a2e606d6f779c5378d6e2dec87b WatchSource:0}: Error finding container f085acc3b661800d9d594f01b15dd01f67f35a2e606d6f779c5378d6e2dec87b: Status 404 returned error can't find the container with id f085acc3b661800d9d594f01b15dd01f67f35a2e606d6f779c5378d6e2dec87b Nov 02 17:53:52.458058 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:52.458036 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p"] Nov 02 17:53:52.569807 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:52.569772 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p" event={"ID":"e7e2e941-5c6e-4a08-913c-e2e56bef424b","Type":"ContainerStarted","Data":"f085acc3b661800d9d594f01b15dd01f67f35a2e606d6f779c5378d6e2dec87b"} Nov 02 17:53:53.405442 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.405412 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:53:53.501409 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.501378 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:53:53.503945 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.503915 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bd25476-7eee-41c9-908c-29c04c64f956-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd25476-7eee-41c9-908c-29c04c64f956-istiod-ca-cert\") pod \"8bd25476-7eee-41c9-908c-29c04c64f956\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " Nov 02 17:53:53.504053 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.503990 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-istio-envoy\") pod \"8bd25476-7eee-41c9-908c-29c04c64f956\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " Nov 02 17:53:53.505285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.504248 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd25476-7eee-41c9-908c-29c04c64f956-kube-api-access-g9gkl\" (UniqueName: \"kubernetes.io/projected/8bd25476-7eee-41c9-908c-29c04c64f956-kube-api-access-g9gkl\") pod \"8bd25476-7eee-41c9-908c-29c04c64f956\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " Nov 02 17:53:53.505285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.504302 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-workload-socket\") pod \"8bd25476-7eee-41c9-908c-29c04c64f956\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " Nov 02 17:53:53.505285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.504340 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd25476-7eee-41c9-908c-29c04c64f956-istio-token\" (UniqueName: \"kubernetes.io/projected/8bd25476-7eee-41c9-908c-29c04c64f956-istio-token\") pod \"8bd25476-7eee-41c9-908c-29c04c64f956\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " Nov 02 17:53:53.505285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.504396 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd25476-7eee-41c9-908c-29c04c64f956-istiod-ca-cert" pod "8bd25476-7eee-41c9-908c-29c04c64f956" (UID: "8bd25476-7eee-41c9-908c-29c04c64f956"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:53.505285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.504522 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-istio-data\") pod \"8bd25476-7eee-41c9-908c-29c04c64f956\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " Nov 02 17:53:53.505285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.504587 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bd25476-7eee-41c9-908c-29c04c64f956-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd25476-7eee-41c9-908c-29c04c64f956-istio-podinfo\") pod \"8bd25476-7eee-41c9-908c-29c04c64f956\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " Nov 02 17:53:53.505285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.504678 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-credential-socket\") pod \"8bd25476-7eee-41c9-908c-29c04c64f956\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " Nov 02 17:53:53.505285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.504742 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-workload-certs\") pod \"8bd25476-7eee-41c9-908c-29c04c64f956\" (UID: \"8bd25476-7eee-41c9-908c-29c04c64f956\") " Nov 02 17:53:53.505285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.504779 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-workload-socket" pod "8bd25476-7eee-41c9-908c-29c04c64f956" (UID: "8bd25476-7eee-41c9-908c-29c04c64f956"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.505285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.505016 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-workload-certs" pod "8bd25476-7eee-41c9-908c-29c04c64f956" (UID: "8bd25476-7eee-41c9-908c-29c04c64f956"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.505926 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.505903 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.506008 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.505952 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd25476-7eee-41c9-908c-29c04c64f956-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.506008 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.505967 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.510186 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.510148 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-istio-data" pod "8bd25476-7eee-41c9-908c-29c04c64f956" (UID: "8bd25476-7eee-41c9-908c-29c04c64f956"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.510186 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.510163 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-credential-socket" pod "8bd25476-7eee-41c9-908c-29c04c64f956" (UID: "8bd25476-7eee-41c9-908c-29c04c64f956"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.514459 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.514436 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-istio-envoy" pod "8bd25476-7eee-41c9-908c-29c04c64f956" (UID: "8bd25476-7eee-41c9-908c-29c04c64f956"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.515316 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.515280 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bd25476-7eee-41c9-908c-29c04c64f956-istio-podinfo" pod "8bd25476-7eee-41c9-908c-29c04c64f956" (UID: "8bd25476-7eee-41c9-908c-29c04c64f956"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:53:53.515884 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.515831 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd25476-7eee-41c9-908c-29c04c64f956-istio-token" pod "8bd25476-7eee-41c9-908c-29c04c64f956" (UID: "8bd25476-7eee-41c9-908c-29c04c64f956"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:53.516820 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.516782 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd25476-7eee-41c9-908c-29c04c64f956-kube-api-access-g9gkl" pod "8bd25476-7eee-41c9-908c-29c04c64f956" (UID: "8bd25476-7eee-41c9-908c-29c04c64f956"). InnerVolumeSpecName "kube-api-access-g9gkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:53.548613 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.548195 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:53:53.554366 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.554343 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:53:53.585184 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.585138 4860 generic.go:356] "Generic (PLEG): container finished" podID="c3ed150c-7f9b-4422-b675-301dcb0c6e3f" containerID="feaa572512091083c7abc3a83f224471f3642af2ec63ca1775b70364353790db" exitCode=0 Nov 02 17:53:53.585388 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.585199 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" Nov 02 17:53:53.585388 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.585254 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" event={"ID":"c3ed150c-7f9b-4422-b675-301dcb0c6e3f","Type":"ContainerDied","Data":"feaa572512091083c7abc3a83f224471f3642af2ec63ca1775b70364353790db"} Nov 02 17:53:53.585388 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.585303 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p" event={"ID":"c3ed150c-7f9b-4422-b675-301dcb0c6e3f","Type":"ContainerDied","Data":"e3e0134169d0cb22a39fcb8d4ff62b04f57fcf09c1b9f75d8ab74aa857d2c4ab"} Nov 02 17:53:53.585388 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.585339 4860 scope.go:122] "RemoveContainer" containerID="feaa572512091083c7abc3a83f224471f3642af2ec63ca1775b70364353790db" Nov 02 17:53:53.587142 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.587118 4860 generic.go:356] "Generic (PLEG): container finished" podID="4db49741-b199-403e-98f4-fbba98d95b84" containerID="9e8497a541e5cf41a39fcbd5046b00f03ae31d270e373f585bb8a04db3da9df5" exitCode=0 Nov 02 17:53:53.587255 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.587177 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" Nov 02 17:53:53.587255 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.587204 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" event={"ID":"4db49741-b199-403e-98f4-fbba98d95b84","Type":"ContainerDied","Data":"9e8497a541e5cf41a39fcbd5046b00f03ae31d270e373f585bb8a04db3da9df5"} Nov 02 17:53:53.587363 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.587263 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns" event={"ID":"4db49741-b199-403e-98f4-fbba98d95b84","Type":"ContainerDied","Data":"e2243171f7b5a5b5d21cc24b74215957b96a208f54c6c1db0684f32d1693ac42"} Nov 02 17:53:53.588932 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.588910 4860 generic.go:356] "Generic (PLEG): container finished" podID="8bd25476-7eee-41c9-908c-29c04c64f956" containerID="1b307520b0bd5961ea1b07b32a774511c62adcc9ae774110d5dcd3c3cb6ebc83" exitCode=0 Nov 02 17:53:53.589014 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.588952 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" Nov 02 17:53:53.589014 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.588993 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" event={"ID":"8bd25476-7eee-41c9-908c-29c04c64f956","Type":"ContainerDied","Data":"1b307520b0bd5961ea1b07b32a774511c62adcc9ae774110d5dcd3c3cb6ebc83"} Nov 02 17:53:53.589171 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.589048 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc" event={"ID":"8bd25476-7eee-41c9-908c-29c04c64f956","Type":"ContainerDied","Data":"4366358493aba3204c361e73a23c1f49b5c596117e4f67a4a20ae3c70e200c88"} Nov 02 17:53:53.590954 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.590929 4860 generic.go:356] "Generic (PLEG): container finished" podID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerID="c282f97739a44629edc3ae9cd598928f8f13e61f1231f7a88b930130e13c5cc3" exitCode=0 Nov 02 17:53:53.591066 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.590991 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" Nov 02 17:53:53.591066 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.591019 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" event={"ID":"48dce99f-6608-4c61-b296-2387ff5d2caa","Type":"ContainerDied","Data":"c282f97739a44629edc3ae9cd598928f8f13e61f1231f7a88b930130e13c5cc3"} Nov 02 17:53:53.591214 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.591094 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p" event={"ID":"48dce99f-6608-4c61-b296-2387ff5d2caa","Type":"ContainerDied","Data":"592c25c66b303743120d902c908618f73d739c70f819d9c80cdd04c22a051821"} Nov 02 17:53:53.594792 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.594763 4860 generic.go:356] "Generic (PLEG): container finished" podID="e7e2e941-5c6e-4a08-913c-e2e56bef424b" containerID="cdbb516d9028519d96d19562e7bae8d5ea0b92ec7fd8f8ea9858bd4dbd2177ed" exitCode=0 Nov 02 17:53:53.594948 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.594841 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p" event={"ID":"e7e2e941-5c6e-4a08-913c-e2e56bef424b","Type":"ContainerDied","Data":"cdbb516d9028519d96d19562e7bae8d5ea0b92ec7fd8f8ea9858bd4dbd2177ed"} Nov 02 17:53:53.601212 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.601193 4860 scope.go:122] "RemoveContainer" containerID="feaa572512091083c7abc3a83f224471f3642af2ec63ca1775b70364353790db" Nov 02 17:53:53.602564 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:53:53.602539 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feaa572512091083c7abc3a83f224471f3642af2ec63ca1775b70364353790db\": container with ID starting with feaa572512091083c7abc3a83f224471f3642af2ec63ca1775b70364353790db not found: ID does not exist" containerID="feaa572512091083c7abc3a83f224471f3642af2ec63ca1775b70364353790db" Nov 02 17:53:53.602670 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.602572 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feaa572512091083c7abc3a83f224471f3642af2ec63ca1775b70364353790db"} err="failed to get container status \"feaa572512091083c7abc3a83f224471f3642af2ec63ca1775b70364353790db\": rpc error: code = NotFound desc = could not find container \"feaa572512091083c7abc3a83f224471f3642af2ec63ca1775b70364353790db\": container with ID starting with feaa572512091083c7abc3a83f224471f3642af2ec63ca1775b70364353790db not found: ID does not exist" Nov 02 17:53:53.602670 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.602595 4860 scope.go:122] "RemoveContainer" containerID="9e8497a541e5cf41a39fcbd5046b00f03ae31d270e373f585bb8a04db3da9df5" Nov 02 17:53:53.607658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.607088 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-istio-envoy\") pod \"4db49741-b199-403e-98f4-fbba98d95b84\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " Nov 02 17:53:53.607658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.607162 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-token\" (UniqueName: \"kubernetes.io/projected/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-token\") pod \"48dce99f-6608-4c61-b296-2387ff5d2caa\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " Nov 02 17:53:53.607658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.607215 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-istio-data\") pod \"4db49741-b199-403e-98f4-fbba98d95b84\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " Nov 02 17:53:53.607658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.607255 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-workload-certs\") pod \"4db49741-b199-403e-98f4-fbba98d95b84\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " Nov 02 17:53:53.607658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.607295 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-data\") pod \"48dce99f-6608-4c61-b296-2387ff5d2caa\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " Nov 02 17:53:53.607658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.607343 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db49741-b199-403e-98f4-fbba98d95b84-istio-token\" (UniqueName: \"kubernetes.io/projected/4db49741-b199-403e-98f4-fbba98d95b84-istio-token\") pod \"4db49741-b199-403e-98f4-fbba98d95b84\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " Nov 02 17:53:53.607658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.607374 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-kube-api-access-gtmqt\" (UniqueName: \"kubernetes.io/projected/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-kube-api-access-gtmqt\") pod \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " Nov 02 17:53:53.607658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.607426 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-workload-socket\") pod \"4db49741-b199-403e-98f4-fbba98d95b84\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " Nov 02 17:53:53.607658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.607470 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-podinfo\") pod \"48dce99f-6608-4c61-b296-2387ff5d2caa\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " Nov 02 17:53:53.607658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.607499 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4db49741-b199-403e-98f4-fbba98d95b84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db49741-b199-403e-98f4-fbba98d95b84-istio-podinfo\") pod \"4db49741-b199-403e-98f4-fbba98d95b84\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " Nov 02 17:53:53.607658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.607536 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-envoy\") pod \"48dce99f-6608-4c61-b296-2387ff5d2caa\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " Nov 02 17:53:53.607658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.607577 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-workload-certs\") pod \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " Nov 02 17:53:53.608833 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.608649 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-istio-data" pod "4db49741-b199-403e-98f4-fbba98d95b84" (UID: "4db49741-b199-403e-98f4-fbba98d95b84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.609539 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.609284 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-workload-certs" pod "c3ed150c-7f9b-4422-b675-301dcb0c6e3f" (UID: "c3ed150c-7f9b-4422-b675-301dcb0c6e3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.611114 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.610225 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-workload-socket" pod "4db49741-b199-403e-98f4-fbba98d95b84" (UID: "4db49741-b199-403e-98f4-fbba98d95b84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.614508 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.614484 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-workload-certs" pod "4db49741-b199-403e-98f4-fbba98d95b84" (UID: "4db49741-b199-403e-98f4-fbba98d95b84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.616063 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.616024 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-data" pod "48dce99f-6608-4c61-b296-2387ff5d2caa" (UID: "48dce99f-6608-4c61-b296-2387ff5d2caa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.616163 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.616057 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-credential-socket\") pod \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " Nov 02 17:53:53.616163 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.616100 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48dce99f-6608-4c61-b296-2387ff5d2caa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dce99f-6608-4c61-b296-2387ff5d2caa-istiod-ca-cert\") pod \"48dce99f-6608-4c61-b296-2387ff5d2caa\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " Nov 02 17:53:53.616163 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.616146 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-podinfo\") pod \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " Nov 02 17:53:53.616342 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.616194 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-workload-certs\") pod \"48dce99f-6608-4c61-b296-2387ff5d2caa\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " Nov 02 17:53:53.616342 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.616234 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-envoy\") pod \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " Nov 02 17:53:53.616342 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.616281 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-data\") pod \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " Nov 02 17:53:53.616509 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.616354 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48dce99f-6608-4c61-b296-2387ff5d2caa-kube-api-access-ndjsx\" (UniqueName: \"kubernetes.io/projected/48dce99f-6608-4c61-b296-2387ff5d2caa-kube-api-access-ndjsx\") pod \"48dce99f-6608-4c61-b296-2387ff5d2caa\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " Nov 02 17:53:53.616509 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.616382 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db49741-b199-403e-98f4-fbba98d95b84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db49741-b199-403e-98f4-fbba98d95b84-istiod-ca-cert\") pod \"4db49741-b199-403e-98f4-fbba98d95b84\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " Nov 02 17:53:53.616509 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.616481 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-token\" (UniqueName: \"kubernetes.io/projected/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-token\") pod \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " Nov 02 17:53:53.617254 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.616526 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-credential-socket" pod "c3ed150c-7f9b-4422-b675-301dcb0c6e3f" (UID: "c3ed150c-7f9b-4422-b675-301dcb0c6e3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.617254 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.617092 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-credential-socket\") pod \"48dce99f-6608-4c61-b296-2387ff5d2caa\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " Nov 02 17:53:53.617254 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.617149 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-workload-socket\") pod \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " Nov 02 17:53:53.617254 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.617216 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-workload-socket\") pod \"48dce99f-6608-4c61-b296-2387ff5d2caa\" (UID: \"48dce99f-6608-4c61-b296-2387ff5d2caa\") " Nov 02 17:53:53.617479 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.617287 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48dce99f-6608-4c61-b296-2387ff5d2caa-istiod-ca-cert" pod "48dce99f-6608-4c61-b296-2387ff5d2caa" (UID: "48dce99f-6608-4c61-b296-2387ff5d2caa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:53.617479 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.617301 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db49741-b199-403e-98f4-fbba98d95b84-kube-api-access-95v29\" (UniqueName: \"kubernetes.io/projected/4db49741-b199-403e-98f4-fbba98d95b84-kube-api-access-95v29\") pod \"4db49741-b199-403e-98f4-fbba98d95b84\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " Nov 02 17:53:53.617479 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.617354 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-credential-socket\") pod \"4db49741-b199-403e-98f4-fbba98d95b84\" (UID: \"4db49741-b199-403e-98f4-fbba98d95b84\") " Nov 02 17:53:53.617479 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.617458 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istiod-ca-cert\") pod \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\" (UID: \"c3ed150c-7f9b-4422-b675-301dcb0c6e3f\") " Nov 02 17:53:53.617691 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.617604 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-data" pod "c3ed150c-7f9b-4422-b675-301dcb0c6e3f" (UID: "c3ed150c-7f9b-4422-b675-301dcb0c6e3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.617743 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.617718 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-workload-socket" pod "c3ed150c-7f9b-4422-b675-301dcb0c6e3f" (UID: "c3ed150c-7f9b-4422-b675-301dcb0c6e3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.619718 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.618258 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-token" pod "48dce99f-6608-4c61-b296-2387ff5d2caa" (UID: "48dce99f-6608-4c61-b296-2387ff5d2caa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:53.619718 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.618318 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db49741-b199-403e-98f4-fbba98d95b84-istiod-ca-cert" pod "4db49741-b199-403e-98f4-fbba98d95b84" (UID: "4db49741-b199-403e-98f4-fbba98d95b84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:53.619718 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.618324 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-workload-certs" pod "48dce99f-6608-4c61-b296-2387ff5d2caa" (UID: "48dce99f-6608-4c61-b296-2387ff5d2caa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.619718 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.618362 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-credential-socket" pod "48dce99f-6608-4c61-b296-2387ff5d2caa" (UID: "48dce99f-6608-4c61-b296-2387ff5d2caa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.619718 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.618535 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-credential-socket" pod "4db49741-b199-403e-98f4-fbba98d95b84" (UID: "4db49741-b199-403e-98f4-fbba98d95b84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.619718 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.618810 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istiod-ca-cert" pod "c3ed150c-7f9b-4422-b675-301dcb0c6e3f" (UID: "c3ed150c-7f9b-4422-b675-301dcb0c6e3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:53.619718 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.618980 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-kube-api-access-gtmqt" pod "c3ed150c-7f9b-4422-b675-301dcb0c6e3f" (UID: "c3ed150c-7f9b-4422-b675-301dcb0c6e3f"). InnerVolumeSpecName "kube-api-access-gtmqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:53.619718 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.619220 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-workload-socket" pod "48dce99f-6608-4c61-b296-2387ff5d2caa" (UID: "48dce99f-6608-4c61-b296-2387ff5d2caa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.619718 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.619629 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd25476-7eee-41c9-908c-29c04c64f956-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.619797 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.619821 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.619835 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.620121 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.620140 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.620166 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.620180 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.620194 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.620208 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dce99f-6608-4c61-b296-2387ff5d2caa-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.620222 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.620234 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.620246 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.620258 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db49741-b199-403e-98f4-fbba98d95b84-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.620270 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9gkl\" (UniqueName: \"kubernetes.io/projected/8bd25476-7eee-41c9-908c-29c04c64f956-kube-api-access-g9gkl\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.620289 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd25476-7eee-41c9-908c-29c04c64f956-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.620301 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.620317 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.620330 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.620817 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.620343 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd25476-7eee-41c9-908c-29c04c64f956-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.621942 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.621902 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4db49741-b199-403e-98f4-fbba98d95b84-istio-podinfo" pod "4db49741-b199-403e-98f4-fbba98d95b84" (UID: "4db49741-b199-403e-98f4-fbba98d95b84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:53:53.627100 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.627003 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-podinfo" pod "48dce99f-6608-4c61-b296-2387ff5d2caa" (UID: "48dce99f-6608-4c61-b296-2387ff5d2caa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:53:53.627100 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.627052 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db49741-b199-403e-98f4-fbba98d95b84-kube-api-access-95v29" pod "4db49741-b199-403e-98f4-fbba98d95b84" (UID: "4db49741-b199-403e-98f4-fbba98d95b84"). InnerVolumeSpecName "kube-api-access-95v29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:53.627260 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.627082 4860 scope.go:122] "RemoveContainer" containerID="9e8497a541e5cf41a39fcbd5046b00f03ae31d270e373f585bb8a04db3da9df5" Nov 02 17:53:53.628687 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:53:53.627518 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e8497a541e5cf41a39fcbd5046b00f03ae31d270e373f585bb8a04db3da9df5\": container with ID starting with 9e8497a541e5cf41a39fcbd5046b00f03ae31d270e373f585bb8a04db3da9df5 not found: ID does not exist" containerID="9e8497a541e5cf41a39fcbd5046b00f03ae31d270e373f585bb8a04db3da9df5" Nov 02 17:53:53.628687 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.627553 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8497a541e5cf41a39fcbd5046b00f03ae31d270e373f585bb8a04db3da9df5"} err="failed to get container status \"9e8497a541e5cf41a39fcbd5046b00f03ae31d270e373f585bb8a04db3da9df5\": rpc error: code = NotFound desc = could not find container \"9e8497a541e5cf41a39fcbd5046b00f03ae31d270e373f585bb8a04db3da9df5\": container with ID starting with 9e8497a541e5cf41a39fcbd5046b00f03ae31d270e373f585bb8a04db3da9df5 not found: ID does not exist" Nov 02 17:53:53.628687 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.627579 4860 scope.go:122] "RemoveContainer" containerID="1b307520b0bd5961ea1b07b32a774511c62adcc9ae774110d5dcd3c3cb6ebc83" Nov 02 17:53:53.630378 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.630346 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db49741-b199-403e-98f4-fbba98d95b84-istio-token" pod "4db49741-b199-403e-98f4-fbba98d95b84" (UID: "4db49741-b199-403e-98f4-fbba98d95b84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:53.631207 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.631181 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48dce99f-6608-4c61-b296-2387ff5d2caa-kube-api-access-ndjsx" pod "48dce99f-6608-4c61-b296-2387ff5d2caa" (UID: "48dce99f-6608-4c61-b296-2387ff5d2caa"). InnerVolumeSpecName "kube-api-access-ndjsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:53.631304 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.631241 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-envoy" pod "48dce99f-6608-4c61-b296-2387ff5d2caa" (UID: "48dce99f-6608-4c61-b296-2387ff5d2caa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.631363 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.631303 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-podinfo" pod "c3ed150c-7f9b-4422-b675-301dcb0c6e3f" (UID: "c3ed150c-7f9b-4422-b675-301dcb0c6e3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:53:53.631484 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.631433 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-token" pod "c3ed150c-7f9b-4422-b675-301dcb0c6e3f" (UID: "c3ed150c-7f9b-4422-b675-301dcb0c6e3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:53.631603 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.631505 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-istio-envoy" pod "4db49741-b199-403e-98f4-fbba98d95b84" (UID: "4db49741-b199-403e-98f4-fbba98d95b84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.633030 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.632996 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-envoy" pod "c3ed150c-7f9b-4422-b675-301dcb0c6e3f" (UID: "c3ed150c-7f9b-4422-b675-301dcb0c6e3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:53:53.642678 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.642402 4860 scope.go:122] "RemoveContainer" containerID="1b307520b0bd5961ea1b07b32a774511c62adcc9ae774110d5dcd3c3cb6ebc83" Nov 02 17:53:53.642800 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:53:53.642767 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b307520b0bd5961ea1b07b32a774511c62adcc9ae774110d5dcd3c3cb6ebc83\": container with ID starting with 1b307520b0bd5961ea1b07b32a774511c62adcc9ae774110d5dcd3c3cb6ebc83 not found: ID does not exist" containerID="1b307520b0bd5961ea1b07b32a774511c62adcc9ae774110d5dcd3c3cb6ebc83" Nov 02 17:53:53.642863 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.642813 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b307520b0bd5961ea1b07b32a774511c62adcc9ae774110d5dcd3c3cb6ebc83"} err="failed to get container status \"1b307520b0bd5961ea1b07b32a774511c62adcc9ae774110d5dcd3c3cb6ebc83\": rpc error: code = NotFound desc = could not find container \"1b307520b0bd5961ea1b07b32a774511c62adcc9ae774110d5dcd3c3cb6ebc83\": container with ID starting with 1b307520b0bd5961ea1b07b32a774511c62adcc9ae774110d5dcd3c3cb6ebc83 not found: ID does not exist" Nov 02 17:53:53.642937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.642837 4860 scope.go:122] "RemoveContainer" containerID="c282f97739a44629edc3ae9cd598928f8f13e61f1231f7a88b930130e13c5cc3" Nov 02 17:53:53.655518 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.655427 4860 scope.go:122] "RemoveContainer" containerID="c282f97739a44629edc3ae9cd598928f8f13e61f1231f7a88b930130e13c5cc3" Nov 02 17:53:53.655839 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:53:53.655804 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c282f97739a44629edc3ae9cd598928f8f13e61f1231f7a88b930130e13c5cc3\": container with ID starting with c282f97739a44629edc3ae9cd598928f8f13e61f1231f7a88b930130e13c5cc3 not found: ID does not exist" containerID="c282f97739a44629edc3ae9cd598928f8f13e61f1231f7a88b930130e13c5cc3" Nov 02 17:53:53.655976 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.655864 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c282f97739a44629edc3ae9cd598928f8f13e61f1231f7a88b930130e13c5cc3"} err="failed to get container status \"c282f97739a44629edc3ae9cd598928f8f13e61f1231f7a88b930130e13c5cc3\": rpc error: code = NotFound desc = could not find container \"c282f97739a44629edc3ae9cd598928f8f13e61f1231f7a88b930130e13c5cc3\": container with ID starting with c282f97739a44629edc3ae9cd598928f8f13e61f1231f7a88b930130e13c5cc3 not found: ID does not exist" Nov 02 17:53:53.721601 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.721565 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.721601 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.721597 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95v29\" (UniqueName: \"kubernetes.io/projected/4db49741-b199-403e-98f4-fbba98d95b84-kube-api-access-95v29\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.721858 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.721613 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.721858 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.721626 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db49741-b199-403e-98f4-fbba98d95b84-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.721952 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.721905 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db49741-b199-403e-98f4-fbba98d95b84-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.721952 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.721920 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtmqt\" (UniqueName: \"kubernetes.io/projected/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-kube-api-access-gtmqt\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.721952 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.721939 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.721952 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.721951 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db49741-b199-403e-98f4-fbba98d95b84-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.722144 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.721959 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dce99f-6608-4c61-b296-2387ff5d2caa-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.722144 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.721968 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.722233 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.722188 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.722233 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.722211 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndjsx\" (UniqueName: \"kubernetes.io/projected/48dce99f-6608-4c61-b296-2387ff5d2caa-kube-api-access-ndjsx\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.722233 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.722221 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ed150c-7f9b-4422-b675-301dcb0c6e3f-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:53.780069 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.780029 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc"] Nov 02 17:53:53.792698 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.792665 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-llcgc"] Nov 02 17:53:53.935861 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.935815 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p"] Nov 02 17:53:53.941344 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.941314 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-dlp8p"] Nov 02 17:53:53.976332 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.976300 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p"] Nov 02 17:53:53.981232 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:53.981204 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-gw54p"] Nov 02 17:53:54.019245 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:54.019213 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns"] Nov 02 17:53:54.025342 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:54.025317 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-hh2ns"] Nov 02 17:53:54.549706 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:54.549676 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" Nov 02 17:53:54.810134 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:54.810112 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p" Nov 02 17:53:54.940741 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:54.940713 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7e2e941-5c6e-4a08-913c-e2e56bef424b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e2e941-5c6e-4a08-913c-e2e56bef424b-maas-api-openshift-ca\") pod \"e7e2e941-5c6e-4a08-913c-e2e56bef424b\" (UID: \"e7e2e941-5c6e-4a08-913c-e2e56bef424b\") " Nov 02 17:53:54.940946 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:54.940815 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7e2e941-5c6e-4a08-913c-e2e56bef424b-kube-api-access-7mnnr\" (UniqueName: \"kubernetes.io/projected/e7e2e941-5c6e-4a08-913c-e2e56bef424b-kube-api-access-7mnnr\") pod \"e7e2e941-5c6e-4a08-913c-e2e56bef424b\" (UID: \"e7e2e941-5c6e-4a08-913c-e2e56bef424b\") " Nov 02 17:53:54.940946 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:54.940868 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7e2e941-5c6e-4a08-913c-e2e56bef424b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e2e941-5c6e-4a08-913c-e2e56bef424b-maas-api-xks-ca\") pod \"e7e2e941-5c6e-4a08-913c-e2e56bef424b\" (UID: \"e7e2e941-5c6e-4a08-913c-e2e56bef424b\") " Nov 02 17:53:54.941083 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:54.941065 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e2e941-5c6e-4a08-913c-e2e56bef424b-maas-api-openshift-ca" pod "e7e2e941-5c6e-4a08-913c-e2e56bef424b" (UID: "e7e2e941-5c6e-4a08-913c-e2e56bef424b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:54.941456 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:54.941436 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e2e941-5c6e-4a08-913c-e2e56bef424b-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:54.942996 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:54.942973 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e2e941-5c6e-4a08-913c-e2e56bef424b-kube-api-access-7mnnr" pod "e7e2e941-5c6e-4a08-913c-e2e56bef424b" (UID: "e7e2e941-5c6e-4a08-913c-e2e56bef424b"). InnerVolumeSpecName "kube-api-access-7mnnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:54.943070 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:54.942991 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7e2e941-5c6e-4a08-913c-e2e56bef424b-maas-api-xks-ca" pod "e7e2e941-5c6e-4a08-913c-e2e56bef424b" (UID: "e7e2e941-5c6e-4a08-913c-e2e56bef424b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:55.043101 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:55.043072 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mnnr\" (UniqueName: \"kubernetes.io/projected/e7e2e941-5c6e-4a08-913c-e2e56bef424b-kube-api-access-7mnnr\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:55.043101 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:55.043099 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e2e941-5c6e-4a08-913c-e2e56bef424b-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:53:55.501064 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:55.501032 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" path="/var/lib/kubelet/pods/48dce99f-6608-4c61-b296-2387ff5d2caa/volumes" Nov 02 17:53:55.501583 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:55.501569 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db49741-b199-403e-98f4-fbba98d95b84" path="/var/lib/kubelet/pods/4db49741-b199-403e-98f4-fbba98d95b84/volumes" Nov 02 17:53:55.502059 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:55.502045 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" path="/var/lib/kubelet/pods/8bd25476-7eee-41c9-908c-29c04c64f956/volumes" Nov 02 17:53:55.502860 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:55.502832 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3ed150c-7f9b-4422-b675-301dcb0c6e3f" path="/var/lib/kubelet/pods/c3ed150c-7f9b-4422-b675-301dcb0c6e3f/volumes" Nov 02 17:53:55.620177 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:55.620150 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p" Nov 02 17:53:55.620582 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:55.620147 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p" event={"ID":"e7e2e941-5c6e-4a08-913c-e2e56bef424b","Type":"ContainerDied","Data":"f085acc3b661800d9d594f01b15dd01f67f35a2e606d6f779c5378d6e2dec87b"} Nov 02 17:53:55.620582 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:53:55.620287 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f085acc3b661800d9d594f01b15dd01f67f35a2e606d6f779c5378d6e2dec87b" Nov 02 17:54:06.285659 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.285618 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8"] Nov 02 17:54:06.286650 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.286629 4860 cpu_manager.go:414] "containerMap: removing container" podUID="c3ed150c-7f9b-4422-b675-301dcb0c6e3f" containerName="istio-proxy" Nov 02 17:54:06.286727 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.286651 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3ed150c-7f9b-4422-b675-301dcb0c6e3f" containerName="istio-proxy" Nov 02 17:54:06.286727 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.286667 4860 cpu_manager.go:414] "containerMap: removing container" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerName="istio-proxy" Nov 02 17:54:06.286727 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.286675 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerName="istio-proxy" Nov 02 17:54:06.286727 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.286694 4860 cpu_manager.go:414] "containerMap: removing container" podUID="e7e2e941-5c6e-4a08-913c-e2e56bef424b" containerName="revoke-keys" Nov 02 17:54:06.286727 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.286701 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7e2e941-5c6e-4a08-913c-e2e56bef424b" containerName="revoke-keys" Nov 02 17:54:06.286727 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.286728 4860 cpu_manager.go:414] "containerMap: removing container" podUID="4db49741-b199-403e-98f4-fbba98d95b84" containerName="istio-proxy" Nov 02 17:54:06.287061 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.286736 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db49741-b199-403e-98f4-fbba98d95b84" containerName="istio-proxy" Nov 02 17:54:06.287061 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.286755 4860 cpu_manager.go:414] "containerMap: removing container" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" containerName="istio-proxy" Nov 02 17:54:06.287061 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.286762 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" containerName="istio-proxy" Nov 02 17:54:06.287061 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.286941 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db49741-b199-403e-98f4-fbba98d95b84" containerName="istio-proxy" Nov 02 17:54:06.287061 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.286955 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd25476-7eee-41c9-908c-29c04c64f956" containerName="istio-proxy" Nov 02 17:54:06.287061 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.286973 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3ed150c-7f9b-4422-b675-301dcb0c6e3f" containerName="istio-proxy" Nov 02 17:54:06.287061 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.286980 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="48dce99f-6608-4c61-b296-2387ff5d2caa" containerName="istio-proxy" Nov 02 17:54:06.287061 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.286992 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7e2e941-5c6e-4a08-913c-e2e56bef424b" containerName="revoke-keys" Nov 02 17:54:06.294235 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.294207 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.328393 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.328361 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.328551 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.328474 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.328551 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.328517 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.328551 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.328543 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.328718 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.328583 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.328718 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.328607 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.328718 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.328647 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8km6\" (UniqueName: \"kubernetes.io/projected/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-kube-api-access-g8km6\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.328718 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.328679 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.329006 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.328723 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.331578 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.331546 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8"] Nov 02 17:54:06.430057 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.430022 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.430216 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.430086 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.430216 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.430121 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.430216 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.430181 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8km6\" (UniqueName: \"kubernetes.io/projected/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-kube-api-access-g8km6\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.430321 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.430226 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.430321 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.430292 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.430438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.430371 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.431003 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.430490 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.431003 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.430520 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.431003 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.430631 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.431003 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.430698 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.431003 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.430801 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.431003 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.431004 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.431361 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.431177 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.432537 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.432504 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.432940 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.432923 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.455350 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.455325 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.455457 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.455377 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8km6\" (UniqueName: \"kubernetes.io/projected/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-kube-api-access-g8km6\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.638513 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.638482 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:06.966062 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.966031 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8"] Nov 02 17:54:06.966955 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:54:06.966916 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7e0db0_409a_4c80_b41c_7dfa0a531cd9.slice/crio-a73182a5032b420f6b5ac99345c68ff69acdfb484a2dd786296e055050636fac WatchSource:0}: Error finding container a73182a5032b420f6b5ac99345c68ff69acdfb484a2dd786296e055050636fac: Status 404 returned error can't find the container with id a73182a5032b420f6b5ac99345c68ff69acdfb484a2dd786296e055050636fac Nov 02 17:54:06.969825 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.969791 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:54:06.969936 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.969894 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:54:06.970009 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:06.969938 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:54:07.484376 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:07.484339 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs"] Nov 02 17:54:07.490898 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:07.490871 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" Nov 02 17:54:07.519244 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:07.519214 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-227833-serving-cert\"" Nov 02 17:54:07.521075 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:07.521043 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs"] Nov 02 17:54:07.548514 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:07.548481 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e780ce-29d1-456e-985c-e3d9d61088eb-metrics-certs\") pod \"maas-api-e2e-api-b-227833-6658b99d67-2pgfs\" (UID: \"82e780ce-29d1-456e-985c-e3d9d61088eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" Nov 02 17:54:07.548514 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:07.548527 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82e780ce-29d1-456e-985c-e3d9d61088eb-maas-api-tls\") pod \"maas-api-e2e-api-b-227833-6658b99d67-2pgfs\" (UID: \"82e780ce-29d1-456e-985c-e3d9d61088eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" Nov 02 17:54:07.548748 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:07.548714 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e780ce-29d1-456e-985c-e3d9d61088eb-openshift-service-ca\") pod \"maas-api-e2e-api-b-227833-6658b99d67-2pgfs\" (UID: \"82e780ce-29d1-456e-985c-e3d9d61088eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" Nov 02 17:54:07.548958 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:07.548934 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n9gn\" (UniqueName: \"kubernetes.io/projected/82e780ce-29d1-456e-985c-e3d9d61088eb-kube-api-access-9n9gn\") pod \"maas-api-e2e-api-b-227833-6658b99d67-2pgfs\" (UID: \"82e780ce-29d1-456e-985c-e3d9d61088eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" Nov 02 17:54:07.650314 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:07.650273 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82e780ce-29d1-456e-985c-e3d9d61088eb-maas-api-tls\") pod \"maas-api-e2e-api-b-227833-6658b99d67-2pgfs\" (UID: \"82e780ce-29d1-456e-985c-e3d9d61088eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" Nov 02 17:54:07.650515 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:07.650390 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e780ce-29d1-456e-985c-e3d9d61088eb-openshift-service-ca\") pod \"maas-api-e2e-api-b-227833-6658b99d67-2pgfs\" (UID: \"82e780ce-29d1-456e-985c-e3d9d61088eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" Nov 02 17:54:07.650515 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:54:07.650426 4860 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-227833-serving-cert: secret "maas-api-e2e-api-b-227833-serving-cert" not found Nov 02 17:54:07.650515 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:54:07.650490 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e780ce-29d1-456e-985c-e3d9d61088eb-maas-api-tls podName:82e780ce-29d1-456e-985c-e3d9d61088eb nodeName:}" failed. No retries permitted until 2026-11-02 17:54:08.150474152 +0000 UTC m=+1466.771833906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82e780ce-29d1-456e-985c-e3d9d61088eb-maas-api-tls") pod "maas-api-e2e-api-b-227833-6658b99d67-2pgfs" (UID: "82e780ce-29d1-456e-985c-e3d9d61088eb") : secret "maas-api-e2e-api-b-227833-serving-cert" not found Nov 02 17:54:07.650688 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:07.650557 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n9gn\" (UniqueName: \"kubernetes.io/projected/82e780ce-29d1-456e-985c-e3d9d61088eb-kube-api-access-9n9gn\") pod \"maas-api-e2e-api-b-227833-6658b99d67-2pgfs\" (UID: \"82e780ce-29d1-456e-985c-e3d9d61088eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" Nov 02 17:54:07.650756 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:07.650734 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e780ce-29d1-456e-985c-e3d9d61088eb-metrics-certs\") pod \"maas-api-e2e-api-b-227833-6658b99d67-2pgfs\" (UID: \"82e780ce-29d1-456e-985c-e3d9d61088eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" Nov 02 17:54:07.651319 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:07.651296 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e780ce-29d1-456e-985c-e3d9d61088eb-openshift-service-ca\") pod \"maas-api-e2e-api-b-227833-6658b99d67-2pgfs\" (UID: \"82e780ce-29d1-456e-985c-e3d9d61088eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" Nov 02 17:54:07.653282 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:07.653257 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e780ce-29d1-456e-985c-e3d9d61088eb-metrics-certs\") pod \"maas-api-e2e-api-b-227833-6658b99d67-2pgfs\" (UID: \"82e780ce-29d1-456e-985c-e3d9d61088eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" Nov 02 17:54:07.687105 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:07.686317 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n9gn\" (UniqueName: \"kubernetes.io/projected/82e780ce-29d1-456e-985c-e3d9d61088eb-kube-api-access-9n9gn\") pod \"maas-api-e2e-api-b-227833-6658b99d67-2pgfs\" (UID: \"82e780ce-29d1-456e-985c-e3d9d61088eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" Nov 02 17:54:07.751882 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:07.751769 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" event={"ID":"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9","Type":"ContainerStarted","Data":"8a3e1c43680b4fb27a42a044d41ff690e46675736f8f14dcb228a6c437d7da1d"} Nov 02 17:54:07.752064 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:07.751944 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" event={"ID":"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9","Type":"ContainerStarted","Data":"a73182a5032b420f6b5ac99345c68ff69acdfb484a2dd786296e055050636fac"} Nov 02 17:54:07.802429 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:07.802370 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" podStartSLOduration=1.8023497210000001 podStartE2EDuration="1.802349721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:07.80046476 +0000 UTC m=+1466.421824540" watchObservedRunningTime="2026-11-02 17:54:07.802349721 +0000 UTC m=+1466.423709500" Nov 02 17:54:08.182692 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:08.182650 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82e780ce-29d1-456e-985c-e3d9d61088eb-maas-api-tls\") pod \"maas-api-e2e-api-b-227833-6658b99d67-2pgfs\" (UID: \"82e780ce-29d1-456e-985c-e3d9d61088eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" Nov 02 17:54:08.185974 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:08.185941 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82e780ce-29d1-456e-985c-e3d9d61088eb-maas-api-tls\") pod \"maas-api-e2e-api-b-227833-6658b99d67-2pgfs\" (UID: \"82e780ce-29d1-456e-985c-e3d9d61088eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" Nov 02 17:54:08.428896 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:08.426431 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" Nov 02 17:54:08.649485 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:08.647954 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:08.649485 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:08.649012 4860 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:54:08.649485 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:08.649064 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" podUID="2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 02 17:54:08.930885 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:08.930142 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs"] Nov 02 17:54:09.638882 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:09.638757 4860 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:54:09.638882 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:09.638824 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" podUID="2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 02 17:54:09.980024 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:09.979891 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" event={"ID":"82e780ce-29d1-456e-985c-e3d9d61088eb","Type":"ContainerStarted","Data":"44b99dc5bfadfe9a7922c208ab963e3036713dddc3f3211c1a2f1be89991000e"} Nov 02 17:54:09.980024 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:09.979959 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" event={"ID":"82e780ce-29d1-456e-985c-e3d9d61088eb","Type":"ContainerStarted","Data":"a8a991d56f7e8899eb233cc56fe4e8c42608ef64c1b15b0a9a4114e7314fbca1"} Nov 02 17:54:09.980024 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:09.979981 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" Nov 02 17:54:10.024156 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:10.024101 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" podStartSLOduration=2.603673075 podStartE2EDuration="3.024087891s" totalImagesPullingTime="420.414816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:10.022154371 +0000 UTC m=+1468.643514146" watchObservedRunningTime="2026-11-02 17:54:10.024087891 +0000 UTC m=+1468.645447666" Nov 02 17:54:10.639628 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:10.639593 4860 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:54:10.639828 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:10.639681 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" podUID="2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 02 17:54:11.640676 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:11.640639 4860 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:54:11.641320 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:11.641273 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" podUID="2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 02 17:54:12.639736 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:12.639698 4860 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:54:12.639935 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:12.639751 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" podUID="2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 02 17:54:13.644928 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:13.644892 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:13.645492 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:13.645181 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:13.646089 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:13.646067 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:54:15.993862 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:15.993821 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" Nov 02 17:54:21.772060 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:54:21.772015 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:54:25.096036 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:25.095944 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-v9l5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:25.096735 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:25.096698 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" podUID="62b05251-7074-41bc-8994-c1051abee475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:34.729310 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:34.729274 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq"] Nov 02 17:54:34.729818 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:34.729539 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" podUID="6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6" containerName="maas-api" containerID="cri-o://77a0f6f12cf12754b1729cddfd287883fef7c95aef59564e5cef37f419ce9895" gracePeriod=30 Nov 02 17:54:35.060530 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.060497 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" Nov 02 17:54:35.211870 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.193357 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-openshift-service-ca\") pod \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\" (UID: \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\") " Nov 02 17:54:35.212239 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.212206 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-maas-api-tls\") pod \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\" (UID: \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\") " Nov 02 17:54:35.212386 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.212372 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-metrics-certs\") pod \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\" (UID: \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\") " Nov 02 17:54:35.212557 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.212542 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-kube-api-access-8jffh\" (UniqueName: \"kubernetes.io/projected/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-kube-api-access-8jffh\") pod \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\" (UID: \"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6\") " Nov 02 17:54:35.223929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.223883 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-kube-api-access-8jffh" pod "6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6" (UID: "6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6"). InnerVolumeSpecName "kube-api-access-8jffh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:35.224530 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.224500 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-openshift-service-ca" pod "6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6" (UID: "6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:35.266997 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.254434 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-maas-api-tls" pod "6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6" (UID: "6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:35.345872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.340562 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:54:35.345872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.340598 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:54:35.345872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.340613 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jffh\" (UniqueName: \"kubernetes.io/projected/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-kube-api-access-8jffh\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:54:35.345872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.341371 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-metrics-certs" pod "6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6" (UID: "6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:35.387998 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.387949 4860 generic.go:356] "Generic (PLEG): container finished" podID="6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6" containerID="77a0f6f12cf12754b1729cddfd287883fef7c95aef59564e5cef37f419ce9895" exitCode=0 Nov 02 17:54:35.389246 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.389130 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" event={"ID":"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6","Type":"ContainerDied","Data":"77a0f6f12cf12754b1729cddfd287883fef7c95aef59564e5cef37f419ce9895"} Nov 02 17:54:35.390166 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.390141 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" event={"ID":"6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6","Type":"ContainerDied","Data":"d4581a173891de6115ac04ed6c7e7e1291e4744b50e54ab605cee6e30f6879d2"} Nov 02 17:54:35.391060 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.391041 4860 scope.go:122] "RemoveContainer" containerID="77a0f6f12cf12754b1729cddfd287883fef7c95aef59564e5cef37f419ce9895" Nov 02 17:54:35.392088 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.392069 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq" Nov 02 17:54:35.432624 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.432601 4860 scope.go:122] "RemoveContainer" containerID="77a0f6f12cf12754b1729cddfd287883fef7c95aef59564e5cef37f419ce9895" Nov 02 17:54:35.433083 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:54:35.433011 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a0f6f12cf12754b1729cddfd287883fef7c95aef59564e5cef37f419ce9895\": container with ID starting with 77a0f6f12cf12754b1729cddfd287883fef7c95aef59564e5cef37f419ce9895 not found: ID does not exist" containerID="77a0f6f12cf12754b1729cddfd287883fef7c95aef59564e5cef37f419ce9895" Nov 02 17:54:35.433083 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.433052 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a0f6f12cf12754b1729cddfd287883fef7c95aef59564e5cef37f419ce9895"} err="failed to get container status \"77a0f6f12cf12754b1729cddfd287883fef7c95aef59564e5cef37f419ce9895\": rpc error: code = NotFound desc = could not find container \"77a0f6f12cf12754b1729cddfd287883fef7c95aef59564e5cef37f419ce9895\": container with ID starting with 77a0f6f12cf12754b1729cddfd287883fef7c95aef59564e5cef37f419ce9895 not found: ID does not exist" Nov 02 17:54:35.442241 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.442181 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:54:35.480635 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.480594 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq"] Nov 02 17:54:35.481929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.481894 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04e682fe-587c4cfb86-vglhq"] Nov 02 17:54:35.518307 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:35.518271 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6" path="/var/lib/kubelet/pods/6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6/volumes" Nov 02 17:54:41.078565 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:41.078526 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm"] Nov 02 17:54:41.796236 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:41.796209 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 17:54:41.803571 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:41.803541 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 17:54:41.810110 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:41.810087 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 17:54:41.820614 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:41.820588 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 17:54:51.796575 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:54:51.796498 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:54:56.102120 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.102067 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" podUID="b9d8921d-5111-4a20-b19c-23cc4cd063fe" containerName="main" containerID="cri-o://8e8f8787befe95609a462b146273c4376c926085821d9165c3dfc249a318fa3f" gracePeriod=45 Nov 02 17:54:56.106456 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.106423 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" podUID="b9d8921d-5111-4a20-b19c-23cc4cd063fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.149:8000/ready\": EOF" Nov 02 17:54:56.404034 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.404005 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:54:56.450271 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.450232 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-tmp-dir\") pod \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " Nov 02 17:54:56.450454 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.450358 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9d8921d-5111-4a20-b19c-23cc4cd063fe-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d8921d-5111-4a20-b19c-23cc4cd063fe-tls-certs\") pod \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " Nov 02 17:54:56.450454 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.450387 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-dshm\") pod \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " Nov 02 17:54:56.450582 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.450483 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-home\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-home\") pod \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " Nov 02 17:54:56.450582 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.450495 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-tmp-dir" pod "b9d8921d-5111-4a20-b19c-23cc4cd063fe" (UID: "b9d8921d-5111-4a20-b19c-23cc4cd063fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:56.450582 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.450539 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-model-cache\") pod \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " Nov 02 17:54:56.450582 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.450568 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d8921d-5111-4a20-b19c-23cc4cd063fe-kube-api-access-pplcs\" (UniqueName: \"kubernetes.io/projected/b9d8921d-5111-4a20-b19c-23cc4cd063fe-kube-api-access-pplcs\") pod \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\" (UID: \"b9d8921d-5111-4a20-b19c-23cc4cd063fe\") " Nov 02 17:54:56.451010 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.450979 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-home" pod "b9d8921d-5111-4a20-b19c-23cc4cd063fe" (UID: "b9d8921d-5111-4a20-b19c-23cc4cd063fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:56.451500 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.451472 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-model-cache" pod "b9d8921d-5111-4a20-b19c-23cc4cd063fe" (UID: "b9d8921d-5111-4a20-b19c-23cc4cd063fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:56.452028 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.452004 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:54:56.452130 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.452030 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:54:56.452130 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.452044 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:54:56.452955 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.452918 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-dshm" pod "b9d8921d-5111-4a20-b19c-23cc4cd063fe" (UID: "b9d8921d-5111-4a20-b19c-23cc4cd063fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:56.453052 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.453027 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9d8921d-5111-4a20-b19c-23cc4cd063fe-tls-certs" pod "b9d8921d-5111-4a20-b19c-23cc4cd063fe" (UID: "b9d8921d-5111-4a20-b19c-23cc4cd063fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:56.453574 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.453545 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d8921d-5111-4a20-b19c-23cc4cd063fe-kube-api-access-pplcs" pod "b9d8921d-5111-4a20-b19c-23cc4cd063fe" (UID: "b9d8921d-5111-4a20-b19c-23cc4cd063fe"). InnerVolumeSpecName "kube-api-access-pplcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:56.553878 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.553825 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pplcs\" (UniqueName: \"kubernetes.io/projected/b9d8921d-5111-4a20-b19c-23cc4cd063fe-kube-api-access-pplcs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:54:56.553878 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.553878 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d8921d-5111-4a20-b19c-23cc4cd063fe-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:54:56.554137 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.553894 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9d8921d-5111-4a20-b19c-23cc4cd063fe-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:54:56.698514 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.698424 4860 generic.go:356] "Generic (PLEG): container finished" podID="b9d8921d-5111-4a20-b19c-23cc4cd063fe" containerID="8e8f8787befe95609a462b146273c4376c926085821d9165c3dfc249a318fa3f" exitCode=0 Nov 02 17:54:56.698514 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.698466 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" Nov 02 17:54:56.698736 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.698505 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" event={"ID":"b9d8921d-5111-4a20-b19c-23cc4cd063fe","Type":"ContainerDied","Data":"8e8f8787befe95609a462b146273c4376c926085821d9165c3dfc249a318fa3f"} Nov 02 17:54:56.698736 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.698557 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm" event={"ID":"b9d8921d-5111-4a20-b19c-23cc4cd063fe","Type":"ContainerDied","Data":"01a4f88cc878dada4dc9cb0681f5f58038e9164355f8d7acac85ed4d9f435e88"} Nov 02 17:54:56.698736 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.698576 4860 scope.go:122] "RemoveContainer" containerID="8e8f8787befe95609a462b146273c4376c926085821d9165c3dfc249a318fa3f" Nov 02 17:54:56.711145 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.711127 4860 scope.go:122] "RemoveContainer" containerID="8e8f8787befe95609a462b146273c4376c926085821d9165c3dfc249a318fa3f" Nov 02 17:54:56.711436 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:54:56.711405 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8f8787befe95609a462b146273c4376c926085821d9165c3dfc249a318fa3f\": container with ID starting with 8e8f8787befe95609a462b146273c4376c926085821d9165c3dfc249a318fa3f not found: ID does not exist" containerID="8e8f8787befe95609a462b146273c4376c926085821d9165c3dfc249a318fa3f" Nov 02 17:54:56.711532 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.711434 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8f8787befe95609a462b146273c4376c926085821d9165c3dfc249a318fa3f"} err="failed to get container status \"8e8f8787befe95609a462b146273c4376c926085821d9165c3dfc249a318fa3f\": rpc error: code = NotFound desc = could not find container \"8e8f8787befe95609a462b146273c4376c926085821d9165c3dfc249a318fa3f\": container with ID starting with 8e8f8787befe95609a462b146273c4376c926085821d9165c3dfc249a318fa3f not found: ID does not exist" Nov 02 17:54:56.752417 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.752360 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm"] Nov 02 17:54:56.764823 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:56.764784 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-04e682fe/e2e-lifecycle-model-04e682fe-kserve-596fd9564f-jvkcm"] Nov 02 17:54:57.502505 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:54:57.502469 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9d8921d-5111-4a20-b19c-23cc4cd063fe" path="/var/lib/kubelet/pods/b9d8921d-5111-4a20-b19c-23cc4cd063fe/volumes" Nov 02 17:55:08.724343 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:08.724306 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf"] Nov 02 17:55:08.725138 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:08.725116 4860 cpu_manager.go:414] "containerMap: removing container" podUID="6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6" containerName="maas-api" Nov 02 17:55:08.725138 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:08.725135 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6" containerName="maas-api" Nov 02 17:55:08.725285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:08.725169 4860 cpu_manager.go:414] "containerMap: removing container" podUID="b9d8921d-5111-4a20-b19c-23cc4cd063fe" containerName="main" Nov 02 17:55:08.725285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:08.725175 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d8921d-5111-4a20-b19c-23cc4cd063fe" containerName="main" Nov 02 17:55:08.725352 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:08.725288 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6214eb11-a46c-49d2-8d4c-dfb6b0fc79c6" containerName="maas-api" Nov 02 17:55:08.725352 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:08.725300 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d8921d-5111-4a20-b19c-23cc4cd063fe" containerName="main" Nov 02 17:55:08.729210 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:08.729189 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:08.756091 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:08.756049 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf"] Nov 02 17:55:08.761277 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:08.761244 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b913b2\"/\"test-model-b913b2-kserve-self-signed-certs\"" Nov 02 17:55:08.761476 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:08.761310 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b913b2\"/\"kube-root-ca.crt\"" Nov 02 17:55:08.761476 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:08.761361 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b913b2\"/\"default-dockercfg-ljztn\"" Nov 02 17:55:08.761476 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:08.761394 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b913b2\"/\"openshift-service-ca.crt\"" Nov 02 17:55:08.900579 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:08.900535 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-home\") pod \"test-model-b913b2-kserve-595f569bfc-jzwrf\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:08.900791 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:08.900588 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-tmp-dir\") pod \"test-model-b913b2-kserve-595f569bfc-jzwrf\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:08.900791 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:08.900719 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vtk9\" (UniqueName: \"kubernetes.io/projected/0281a566-f641-4a84-aa07-03f6e6845bc1-kube-api-access-7vtk9\") pod \"test-model-b913b2-kserve-595f569bfc-jzwrf\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:08.900934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:08.900830 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0281a566-f641-4a84-aa07-03f6e6845bc1-tls-certs\") pod \"test-model-b913b2-kserve-595f569bfc-jzwrf\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:08.901055 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:08.901034 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-model-cache\") pod \"test-model-b913b2-kserve-595f569bfc-jzwrf\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:08.901159 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:08.901122 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-dshm\") pod \"test-model-b913b2-kserve-595f569bfc-jzwrf\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:09.012018 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:09.002661 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-model-cache\") pod \"test-model-b913b2-kserve-595f569bfc-jzwrf\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:09.012018 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:09.003262 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-model-cache\") pod \"test-model-b913b2-kserve-595f569bfc-jzwrf\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:09.012018 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:09.003456 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-dshm\") pod \"test-model-b913b2-kserve-595f569bfc-jzwrf\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:09.012018 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:09.004067 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-home\") pod \"test-model-b913b2-kserve-595f569bfc-jzwrf\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:09.012018 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:09.004441 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-home\") pod \"test-model-b913b2-kserve-595f569bfc-jzwrf\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:09.012018 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:09.004513 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-tmp-dir\") pod \"test-model-b913b2-kserve-595f569bfc-jzwrf\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:09.012018 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:09.004882 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-tmp-dir\") pod \"test-model-b913b2-kserve-595f569bfc-jzwrf\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:09.012018 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:09.004977 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vtk9\" (UniqueName: \"kubernetes.io/projected/0281a566-f641-4a84-aa07-03f6e6845bc1-kube-api-access-7vtk9\") pod \"test-model-b913b2-kserve-595f569bfc-jzwrf\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:09.012018 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:09.006064 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0281a566-f641-4a84-aa07-03f6e6845bc1-tls-certs\") pod \"test-model-b913b2-kserve-595f569bfc-jzwrf\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:09.034762 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:09.028931 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0281a566-f641-4a84-aa07-03f6e6845bc1-tls-certs\") pod \"test-model-b913b2-kserve-595f569bfc-jzwrf\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:09.034762 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:09.029167 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-dshm\") pod \"test-model-b913b2-kserve-595f569bfc-jzwrf\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:09.047789 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:09.047760 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vtk9\" (UniqueName: \"kubernetes.io/projected/0281a566-f641-4a84-aa07-03f6e6845bc1-kube-api-access-7vtk9\") pod \"test-model-b913b2-kserve-595f569bfc-jzwrf\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:09.055884 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:09.050174 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:09.474427 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:09.474395 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf"] Nov 02 17:55:09.479949 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:55:09.475036 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0281a566_f641_4a84_aa07_03f6e6845bc1.slice/crio-d3d5185dbec40c1e305eabc9f957f1c6b08d41747e3019679af6428d66c2bea7 WatchSource:0}: Error finding container d3d5185dbec40c1e305eabc9f957f1c6b08d41747e3019679af6428d66c2bea7: Status 404 returned error can't find the container with id d3d5185dbec40c1e305eabc9f957f1c6b08d41747e3019679af6428d66c2bea7 Nov 02 17:55:09.852869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:09.852303 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" event={"ID":"0281a566-f641-4a84-aa07-03f6e6845bc1","Type":"ContainerStarted","Data":"93e924edf3f941dbf758f5ce5e7ffab8586236d1ed6018e8ab1dfeb23fa305f8"} Nov 02 17:55:09.852869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:09.852356 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" event={"ID":"0281a566-f641-4a84-aa07-03f6e6845bc1","Type":"ContainerStarted","Data":"d3d5185dbec40c1e305eabc9f957f1c6b08d41747e3019679af6428d66c2bea7"} Nov 02 17:55:09.899300 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:09.899245 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" podStartSLOduration=1.899229687 podStartE2EDuration="1.899229687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:09.895118851 +0000 UTC m=+1528.516478628" watchObservedRunningTime="2026-11-02 17:55:09.899229687 +0000 UTC m=+1528.520589492" Nov 02 17:55:10.051869 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:10.051091 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:12.632753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:12.632717 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p"] Nov 02 17:55:12.645382 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:12.645321 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e5fcb-bgg6p"] Nov 02 17:55:13.501587 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:13.501551 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7e2e941-5c6e-4a08-913c-e2e56bef424b" path="/var/lib/kubelet/pods/e7e2e941-5c6e-4a08-913c-e2e56bef424b/volumes" Nov 02 17:55:18.392623 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:18.392589 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp"] Nov 02 17:55:18.393086 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:18.392789 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" podUID="5a72d837-5288-4398-8bd5-3f4b9db3002f" containerName="istio-proxy" containerID="cri-o://952e39c2d290df01b3c1afbb93d9ea5256dd327962c75ead5d81bb19426d0197" gracePeriod=30 Nov 02 17:55:19.050616 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:19.050561 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:19.066118 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:19.066090 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:19.989338 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:19.989307 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 17:55:21.772368 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:55:21.772332 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:55:23.873143 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.873112 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:55:23.913251 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.913217 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a72d837-5288-4398-8bd5-3f4b9db3002f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a72d837-5288-4398-8bd5-3f4b9db3002f-istiod-ca-cert\") pod \"5a72d837-5288-4398-8bd5-3f4b9db3002f\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " Nov 02 17:55:23.913251 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.913258 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-podinfo\") pod \"5a72d837-5288-4398-8bd5-3f4b9db3002f\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " Nov 02 17:55:23.913489 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.913337 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a72d837-5288-4398-8bd5-3f4b9db3002f-kube-api-access-fj4d2\" (UniqueName: \"kubernetes.io/projected/5a72d837-5288-4398-8bd5-3f4b9db3002f-kube-api-access-fj4d2\") pod \"5a72d837-5288-4398-8bd5-3f4b9db3002f\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " Nov 02 17:55:23.913489 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.913380 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-data\") pod \"5a72d837-5288-4398-8bd5-3f4b9db3002f\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " Nov 02 17:55:23.913619 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.913495 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-workload-certs\") pod \"5a72d837-5288-4398-8bd5-3f4b9db3002f\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " Nov 02 17:55:23.913619 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.913523 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-workload-socket\") pod \"5a72d837-5288-4398-8bd5-3f4b9db3002f\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " Nov 02 17:55:23.913619 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.913586 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-token\" (UniqueName: \"kubernetes.io/projected/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-token\") pod \"5a72d837-5288-4398-8bd5-3f4b9db3002f\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " Nov 02 17:55:23.913776 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.913627 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-credential-socket\") pod \"5a72d837-5288-4398-8bd5-3f4b9db3002f\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " Nov 02 17:55:23.913776 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.913661 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a72d837-5288-4398-8bd5-3f4b9db3002f-istiod-ca-cert" pod "5a72d837-5288-4398-8bd5-3f4b9db3002f" (UID: "5a72d837-5288-4398-8bd5-3f4b9db3002f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:23.913776 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.913726 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-envoy\") pod \"5a72d837-5288-4398-8bd5-3f4b9db3002f\" (UID: \"5a72d837-5288-4398-8bd5-3f4b9db3002f\") " Nov 02 17:55:23.913981 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.913823 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-workload-socket" pod "5a72d837-5288-4398-8bd5-3f4b9db3002f" (UID: "5a72d837-5288-4398-8bd5-3f4b9db3002f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:23.914647 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.914362 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-credential-socket" pod "5a72d837-5288-4398-8bd5-3f4b9db3002f" (UID: "5a72d837-5288-4398-8bd5-3f4b9db3002f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:23.914647 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.914419 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-workload-certs" pod "5a72d837-5288-4398-8bd5-3f4b9db3002f" (UID: "5a72d837-5288-4398-8bd5-3f4b9db3002f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:23.914647 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.914441 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a72d837-5288-4398-8bd5-3f4b9db3002f-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:23.914647 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.914464 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:23.914957 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.914657 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-data" pod "5a72d837-5288-4398-8bd5-3f4b9db3002f" (UID: "5a72d837-5288-4398-8bd5-3f4b9db3002f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:23.916350 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.916322 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-podinfo" pod "5a72d837-5288-4398-8bd5-3f4b9db3002f" (UID: "5a72d837-5288-4398-8bd5-3f4b9db3002f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:55:23.916525 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.916508 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a72d837-5288-4398-8bd5-3f4b9db3002f-kube-api-access-fj4d2" pod "5a72d837-5288-4398-8bd5-3f4b9db3002f" (UID: "5a72d837-5288-4398-8bd5-3f4b9db3002f"). InnerVolumeSpecName "kube-api-access-fj4d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:23.916645 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.916628 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-token" pod "5a72d837-5288-4398-8bd5-3f4b9db3002f" (UID: "5a72d837-5288-4398-8bd5-3f4b9db3002f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:23.916751 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:23.916730 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-envoy" pod "5a72d837-5288-4398-8bd5-3f4b9db3002f" (UID: "5a72d837-5288-4398-8bd5-3f4b9db3002f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:24.020087 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:24.019996 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj4d2\" (UniqueName: \"kubernetes.io/projected/5a72d837-5288-4398-8bd5-3f4b9db3002f-kube-api-access-fj4d2\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:24.020087 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:24.020033 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:24.020087 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:24.020047 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:24.020087 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:24.020060 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:24.020087 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:24.020073 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:24.020087 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:24.020086 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:24.020400 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:24.020099 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a72d837-5288-4398-8bd5-3f4b9db3002f-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:24.026079 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:24.026028 4860 generic.go:356] "Generic (PLEG): container finished" podID="5a72d837-5288-4398-8bd5-3f4b9db3002f" containerID="952e39c2d290df01b3c1afbb93d9ea5256dd327962c75ead5d81bb19426d0197" exitCode=0 Nov 02 17:55:24.026255 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:24.026121 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" Nov 02 17:55:24.026255 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:24.026145 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" event={"ID":"5a72d837-5288-4398-8bd5-3f4b9db3002f","Type":"ContainerDied","Data":"952e39c2d290df01b3c1afbb93d9ea5256dd327962c75ead5d81bb19426d0197"} Nov 02 17:55:24.026255 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:24.026199 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp" event={"ID":"5a72d837-5288-4398-8bd5-3f4b9db3002f","Type":"ContainerDied","Data":"a2ae21d43b08b519d5cfee0bc3105c28dbbd9367a6603d56bf904e5f9c10cf9a"} Nov 02 17:55:24.026255 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:24.026228 4860 scope.go:122] "RemoveContainer" containerID="952e39c2d290df01b3c1afbb93d9ea5256dd327962c75ead5d81bb19426d0197" Nov 02 17:55:24.048208 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:24.048111 4860 scope.go:122] "RemoveContainer" containerID="952e39c2d290df01b3c1afbb93d9ea5256dd327962c75ead5d81bb19426d0197" Nov 02 17:55:24.048573 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:55:24.048544 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"952e39c2d290df01b3c1afbb93d9ea5256dd327962c75ead5d81bb19426d0197\": container with ID starting with 952e39c2d290df01b3c1afbb93d9ea5256dd327962c75ead5d81bb19426d0197 not found: ID does not exist" containerID="952e39c2d290df01b3c1afbb93d9ea5256dd327962c75ead5d81bb19426d0197" Nov 02 17:55:24.048724 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:24.048700 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952e39c2d290df01b3c1afbb93d9ea5256dd327962c75ead5d81bb19426d0197"} err="failed to get container status \"952e39c2d290df01b3c1afbb93d9ea5256dd327962c75ead5d81bb19426d0197\": rpc error: code = NotFound desc = could not find container \"952e39c2d290df01b3c1afbb93d9ea5256dd327962c75ead5d81bb19426d0197\": container with ID starting with 952e39c2d290df01b3c1afbb93d9ea5256dd327962c75ead5d81bb19426d0197 not found: ID does not exist" Nov 02 17:55:24.126930 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:24.119156 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp"] Nov 02 17:55:24.126930 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:24.126291 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-48sdp"] Nov 02 17:55:25.588491 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:25.588445 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a72d837-5288-4398-8bd5-3f4b9db3002f" path="/var/lib/kubelet/pods/5a72d837-5288-4398-8bd5-3f4b9db3002f/volumes" Nov 02 17:55:37.462502 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.462467 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5"] Nov 02 17:55:37.464893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.463243 4860 cpu_manager.go:414] "containerMap: removing container" podUID="5a72d837-5288-4398-8bd5-3f4b9db3002f" containerName="istio-proxy" Nov 02 17:55:37.464893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.463260 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a72d837-5288-4398-8bd5-3f4b9db3002f" containerName="istio-proxy" Nov 02 17:55:37.464893 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.463381 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a72d837-5288-4398-8bd5-3f4b9db3002f" containerName="istio-proxy" Nov 02 17:55:37.466942 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.466925 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" Nov 02 17:55:37.490038 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.490013 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5"] Nov 02 17:55:37.497004 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.496708 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a5c1da-serving-cert\"" Nov 02 17:55:37.535464 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.535429 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/719f739a-813b-4f51-9bd4-e523bba953af-maas-api-tls\") pod \"maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5\" (UID: \"719f739a-813b-4f51-9bd4-e523bba953af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" Nov 02 17:55:37.535682 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.535523 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ffv7\" (UniqueName: \"kubernetes.io/projected/719f739a-813b-4f51-9bd4-e523bba953af-kube-api-access-6ffv7\") pod \"maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5\" (UID: \"719f739a-813b-4f51-9bd4-e523bba953af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" Nov 02 17:55:37.535682 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.535646 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/719f739a-813b-4f51-9bd4-e523bba953af-metrics-certs\") pod \"maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5\" (UID: \"719f739a-813b-4f51-9bd4-e523bba953af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" Nov 02 17:55:37.535832 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.535707 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/719f739a-813b-4f51-9bd4-e523bba953af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5\" (UID: \"719f739a-813b-4f51-9bd4-e523bba953af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" Nov 02 17:55:37.636563 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.636525 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/719f739a-813b-4f51-9bd4-e523bba953af-maas-api-tls\") pod \"maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5\" (UID: \"719f739a-813b-4f51-9bd4-e523bba953af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" Nov 02 17:55:37.636734 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.636597 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ffv7\" (UniqueName: \"kubernetes.io/projected/719f739a-813b-4f51-9bd4-e523bba953af-kube-api-access-6ffv7\") pod \"maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5\" (UID: \"719f739a-813b-4f51-9bd4-e523bba953af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" Nov 02 17:55:37.636734 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.636668 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/719f739a-813b-4f51-9bd4-e523bba953af-metrics-certs\") pod \"maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5\" (UID: \"719f739a-813b-4f51-9bd4-e523bba953af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" Nov 02 17:55:37.636734 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.636713 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/719f739a-813b-4f51-9bd4-e523bba953af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5\" (UID: \"719f739a-813b-4f51-9bd4-e523bba953af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" Nov 02 17:55:37.637365 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.637343 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/719f739a-813b-4f51-9bd4-e523bba953af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5\" (UID: \"719f739a-813b-4f51-9bd4-e523bba953af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" Nov 02 17:55:37.639119 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.639099 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/719f739a-813b-4f51-9bd4-e523bba953af-maas-api-tls\") pod \"maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5\" (UID: \"719f739a-813b-4f51-9bd4-e523bba953af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" Nov 02 17:55:37.639224 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.639124 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/719f739a-813b-4f51-9bd4-e523bba953af-metrics-certs\") pod \"maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5\" (UID: \"719f739a-813b-4f51-9bd4-e523bba953af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" Nov 02 17:55:37.661555 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.661531 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ffv7\" (UniqueName: \"kubernetes.io/projected/719f739a-813b-4f51-9bd4-e523bba953af-kube-api-access-6ffv7\") pod \"maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5\" (UID: \"719f739a-813b-4f51-9bd4-e523bba953af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" Nov 02 17:55:37.782086 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:37.782002 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" Nov 02 17:55:38.112658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:38.111775 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5"] Nov 02 17:55:38.119336 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:55:38.119301 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719f739a_813b_4f51_9bd4_e523bba953af.slice/crio-5a4fd547dacfffac71541aed9b417023faf2018b8a8ebfbc0d1176c6b8894f8f WatchSource:0}: Error finding container 5a4fd547dacfffac71541aed9b417023faf2018b8a8ebfbc0d1176c6b8894f8f: Status 404 returned error can't find the container with id 5a4fd547dacfffac71541aed9b417023faf2018b8a8ebfbc0d1176c6b8894f8f Nov 02 17:55:38.163183 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:38.163142 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" event={"ID":"719f739a-813b-4f51-9bd4-e523bba953af","Type":"ContainerStarted","Data":"5a4fd547dacfffac71541aed9b417023faf2018b8a8ebfbc0d1176c6b8894f8f"} Nov 02 17:55:39.176894 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:39.176833 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" event={"ID":"719f739a-813b-4f51-9bd4-e523bba953af","Type":"ContainerStarted","Data":"6058ad0ba329ef2d321c271101c9de9309f3590be6840a815ba5d495c73ca212"} Nov 02 17:55:39.177407 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:39.177001 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" Nov 02 17:55:39.246835 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:39.246773 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" podStartSLOduration=1.9324824760000001 podStartE2EDuration="2.246756547s" totalImagesPullingTime="314.274071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:39.242557938 +0000 UTC m=+1557.863917715" watchObservedRunningTime="2026-11-02 17:55:39.246756547 +0000 UTC m=+1557.868116325" Nov 02 17:55:45.190891 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:45.190845 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" Nov 02 17:55:48.520983 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:48.520943 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd"] Nov 02 17:55:48.521501 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:48.521189 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" podUID="49146668-6dbb-4df5-a1c1-b2c150a74979" containerName="istio-proxy" containerID="cri-o://99f2acd3cb7d3cdb010c6a333b6fa60263c0b73440eb502805ea70e952e02461" gracePeriod=30 Nov 02 17:55:48.592259 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:48.592217 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b"] Nov 02 17:55:48.592460 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:48.592433 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" containerID="cri-o://4945f033c3321035a66c43ef93a2633e49e63f0ec364c4b5dd27fd9d431f979e" gracePeriod=30 Nov 02 17:55:49.264746 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:49.264715 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq"] Nov 02 17:55:49.264987 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:49.264960 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" podUID="57a3d284-8cce-4c08-830a-605040cfb458" containerName="istio-proxy" containerID="cri-o://189d517dfe3103337930a18e894cf7c35cba210420c6375c0c54fc085592bd96" gracePeriod=30 Nov 02 17:55:50.009919 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:50.009879 4860 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:55:50.010306 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:50.009931 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:55:51.772086 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:55:51.772044 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:55:53.893285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.893258 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:55:53.951535 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.951502 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-workload-socket\") pod \"49146668-6dbb-4df5-a1c1-b2c150a74979\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " Nov 02 17:55:53.951689 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.951579 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-workload-certs\") pod \"49146668-6dbb-4df5-a1c1-b2c150a74979\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " Nov 02 17:55:53.951689 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.951618 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49146668-6dbb-4df5-a1c1-b2c150a74979-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49146668-6dbb-4df5-a1c1-b2c150a74979-istiod-ca-cert\") pod \"49146668-6dbb-4df5-a1c1-b2c150a74979\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " Nov 02 17:55:53.951689 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.951655 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-credential-socket\") pod \"49146668-6dbb-4df5-a1c1-b2c150a74979\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " Nov 02 17:55:53.951880 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.951766 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-workload-socket" pod "49146668-6dbb-4df5-a1c1-b2c150a74979" (UID: "49146668-6dbb-4df5-a1c1-b2c150a74979"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:53.952057 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.952036 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-credential-socket" pod "49146668-6dbb-4df5-a1c1-b2c150a74979" (UID: "49146668-6dbb-4df5-a1c1-b2c150a74979"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:53.952174 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.952148 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-workload-certs" pod "49146668-6dbb-4df5-a1c1-b2c150a74979" (UID: "49146668-6dbb-4df5-a1c1-b2c150a74979"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:53.952739 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.952340 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49146668-6dbb-4df5-a1c1-b2c150a74979-istiod-ca-cert" pod "49146668-6dbb-4df5-a1c1-b2c150a74979" (UID: "49146668-6dbb-4df5-a1c1-b2c150a74979"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:53.952739 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.952349 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49146668-6dbb-4df5-a1c1-b2c150a74979-kube-api-access-gwj74\" (UniqueName: \"kubernetes.io/projected/49146668-6dbb-4df5-a1c1-b2c150a74979-kube-api-access-gwj74\") pod \"49146668-6dbb-4df5-a1c1-b2c150a74979\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " Nov 02 17:55:53.952739 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.952416 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-data\") pod \"49146668-6dbb-4df5-a1c1-b2c150a74979\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " Nov 02 17:55:53.952739 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.952437 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-podinfo\") pod \"49146668-6dbb-4df5-a1c1-b2c150a74979\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " Nov 02 17:55:53.952739 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.952459 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-token\" (UniqueName: \"kubernetes.io/projected/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-token\") pod \"49146668-6dbb-4df5-a1c1-b2c150a74979\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " Nov 02 17:55:53.952739 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.952534 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-envoy\") pod \"49146668-6dbb-4df5-a1c1-b2c150a74979\" (UID: \"49146668-6dbb-4df5-a1c1-b2c150a74979\") " Nov 02 17:55:53.952739 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.952696 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-data" pod "49146668-6dbb-4df5-a1c1-b2c150a74979" (UID: "49146668-6dbb-4df5-a1c1-b2c150a74979"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:53.953605 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.953584 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:53.953695 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.953608 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49146668-6dbb-4df5-a1c1-b2c150a74979-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:53.953695 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.953624 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:53.953695 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.953636 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:53.953695 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.953649 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:53.955007 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.954978 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-token" pod "49146668-6dbb-4df5-a1c1-b2c150a74979" (UID: "49146668-6dbb-4df5-a1c1-b2c150a74979"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:53.955129 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.955107 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-podinfo" pod "49146668-6dbb-4df5-a1c1-b2c150a74979" (UID: "49146668-6dbb-4df5-a1c1-b2c150a74979"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:55:53.955232 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.955214 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49146668-6dbb-4df5-a1c1-b2c150a74979-kube-api-access-gwj74" pod "49146668-6dbb-4df5-a1c1-b2c150a74979" (UID: "49146668-6dbb-4df5-a1c1-b2c150a74979"). InnerVolumeSpecName "kube-api-access-gwj74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:53.955716 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.955692 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-envoy" pod "49146668-6dbb-4df5-a1c1-b2c150a74979" (UID: "49146668-6dbb-4df5-a1c1-b2c150a74979"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:53.963304 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:53.963277 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:55:54.054539 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.054504 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-workload-certs\") pod \"20b1a14b-2879-4835-acac-cfa13510175f\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " Nov 02 17:55:54.054759 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.054734 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20b1a14b-2879-4835-acac-cfa13510175f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b1a14b-2879-4835-acac-cfa13510175f-istio-podinfo\") pod \"20b1a14b-2879-4835-acac-cfa13510175f\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " Nov 02 17:55:54.054926 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.054830 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b1a14b-2879-4835-acac-cfa13510175f-istio-token\" (UniqueName: \"kubernetes.io/projected/20b1a14b-2879-4835-acac-cfa13510175f-istio-token\") pod \"20b1a14b-2879-4835-acac-cfa13510175f\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " Nov 02 17:55:54.054926 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.054889 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-credential-socket\") pod \"20b1a14b-2879-4835-acac-cfa13510175f\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " Nov 02 17:55:54.054926 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.054920 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-istio-envoy\") pod \"20b1a14b-2879-4835-acac-cfa13510175f\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " Nov 02 17:55:54.055128 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.054921 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-workload-certs" pod "20b1a14b-2879-4835-acac-cfa13510175f" (UID: "20b1a14b-2879-4835-acac-cfa13510175f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:54.055128 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.054975 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b1a14b-2879-4835-acac-cfa13510175f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b1a14b-2879-4835-acac-cfa13510175f-istiod-ca-cert\") pod \"20b1a14b-2879-4835-acac-cfa13510175f\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " Nov 02 17:55:54.055128 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.055045 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b1a14b-2879-4835-acac-cfa13510175f-kube-api-access-bkzzn\" (UniqueName: \"kubernetes.io/projected/20b1a14b-2879-4835-acac-cfa13510175f-kube-api-access-bkzzn\") pod \"20b1a14b-2879-4835-acac-cfa13510175f\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " Nov 02 17:55:54.055128 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.055066 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-workload-socket\") pod \"20b1a14b-2879-4835-acac-cfa13510175f\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " Nov 02 17:55:54.055128 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.055122 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-istio-data\") pod \"20b1a14b-2879-4835-acac-cfa13510175f\" (UID: \"20b1a14b-2879-4835-acac-cfa13510175f\") " Nov 02 17:55:54.055830 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.055809 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.056021 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.055999 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwj74\" (UniqueName: \"kubernetes.io/projected/49146668-6dbb-4df5-a1c1-b2c150a74979-kube-api-access-gwj74\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.056139 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.056125 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.056237 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.056224 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.056331 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.056320 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49146668-6dbb-4df5-a1c1-b2c150a74979-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.056406 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.055888 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-credential-socket" pod "20b1a14b-2879-4835-acac-cfa13510175f" (UID: "20b1a14b-2879-4835-acac-cfa13510175f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:54.056481 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.055937 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-workload-socket" pod "20b1a14b-2879-4835-acac-cfa13510175f" (UID: "20b1a14b-2879-4835-acac-cfa13510175f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:54.056561 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.055967 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-istio-data" pod "20b1a14b-2879-4835-acac-cfa13510175f" (UID: "20b1a14b-2879-4835-acac-cfa13510175f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:54.056634 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.056311 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b1a14b-2879-4835-acac-cfa13510175f-istiod-ca-cert" pod "20b1a14b-2879-4835-acac-cfa13510175f" (UID: "20b1a14b-2879-4835-acac-cfa13510175f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:54.058371 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.058340 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b1a14b-2879-4835-acac-cfa13510175f-istio-token" pod "20b1a14b-2879-4835-acac-cfa13510175f" (UID: "20b1a14b-2879-4835-acac-cfa13510175f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:54.059144 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.059108 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b1a14b-2879-4835-acac-cfa13510175f-kube-api-access-bkzzn" pod "20b1a14b-2879-4835-acac-cfa13510175f" (UID: "20b1a14b-2879-4835-acac-cfa13510175f"). InnerVolumeSpecName "kube-api-access-bkzzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:54.059320 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.059297 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-istio-envoy" pod "20b1a14b-2879-4835-acac-cfa13510175f" (UID: "20b1a14b-2879-4835-acac-cfa13510175f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:54.059402 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.059355 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20b1a14b-2879-4835-acac-cfa13510175f-istio-podinfo" pod "20b1a14b-2879-4835-acac-cfa13510175f" (UID: "20b1a14b-2879-4835-acac-cfa13510175f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:55:54.157401 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.157364 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b1a14b-2879-4835-acac-cfa13510175f-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.157401 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.157399 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b1a14b-2879-4835-acac-cfa13510175f-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.157401 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.157409 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.157680 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.157419 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.157680 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.157428 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b1a14b-2879-4835-acac-cfa13510175f-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.157680 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.157436 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkzzn\" (UniqueName: \"kubernetes.io/projected/20b1a14b-2879-4835-acac-cfa13510175f-kube-api-access-bkzzn\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.157680 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.157446 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.157680 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.157454 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b1a14b-2879-4835-acac-cfa13510175f-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.306246 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.306187 4860 generic.go:356] "Generic (PLEG): container finished" podID="20b1a14b-2879-4835-acac-cfa13510175f" containerID="4945f033c3321035a66c43ef93a2633e49e63f0ec364c4b5dd27fd9d431f979e" exitCode=0 Nov 02 17:55:54.306366 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.306252 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" Nov 02 17:55:54.306366 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.306276 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" event={"ID":"20b1a14b-2879-4835-acac-cfa13510175f","Type":"ContainerDied","Data":"4945f033c3321035a66c43ef93a2633e49e63f0ec364c4b5dd27fd9d431f979e"} Nov 02 17:55:54.306366 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.306340 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b" event={"ID":"20b1a14b-2879-4835-acac-cfa13510175f","Type":"ContainerDied","Data":"4f1c318afc094c86b6536c8bc07f8678f766e69a23af2785d728119ed0cb6f16"} Nov 02 17:55:54.306536 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.306372 4860 scope.go:122] "RemoveContainer" containerID="4945f033c3321035a66c43ef93a2633e49e63f0ec364c4b5dd27fd9d431f979e" Nov 02 17:55:54.308292 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.308268 4860 generic.go:356] "Generic (PLEG): container finished" podID="49146668-6dbb-4df5-a1c1-b2c150a74979" containerID="99f2acd3cb7d3cdb010c6a333b6fa60263c0b73440eb502805ea70e952e02461" exitCode=0 Nov 02 17:55:54.308434 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.308375 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" Nov 02 17:55:54.308522 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.308502 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" event={"ID":"49146668-6dbb-4df5-a1c1-b2c150a74979","Type":"ContainerDied","Data":"99f2acd3cb7d3cdb010c6a333b6fa60263c0b73440eb502805ea70e952e02461"} Nov 02 17:55:54.308586 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.308551 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd" event={"ID":"49146668-6dbb-4df5-a1c1-b2c150a74979","Type":"ContainerDied","Data":"1258e6fa1f38e52fd78e62da85e710b6ffaebe551437e6ffcc027f1596d14a04"} Nov 02 17:55:54.311177 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.311151 4860 generic.go:356] "Generic (PLEG): container finished" podID="57a3d284-8cce-4c08-830a-605040cfb458" containerID="189d517dfe3103337930a18e894cf7c35cba210420c6375c0c54fc085592bd96" exitCode=0 Nov 02 17:55:54.311302 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.311209 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" event={"ID":"57a3d284-8cce-4c08-830a-605040cfb458","Type":"ContainerDied","Data":"189d517dfe3103337930a18e894cf7c35cba210420c6375c0c54fc085592bd96"} Nov 02 17:55:54.331964 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.331934 4860 scope.go:122] "RemoveContainer" containerID="4945f033c3321035a66c43ef93a2633e49e63f0ec364c4b5dd27fd9d431f979e" Nov 02 17:55:54.335158 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:55:54.335115 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4945f033c3321035a66c43ef93a2633e49e63f0ec364c4b5dd27fd9d431f979e\": container with ID starting with 4945f033c3321035a66c43ef93a2633e49e63f0ec364c4b5dd27fd9d431f979e not found: ID does not exist" containerID="4945f033c3321035a66c43ef93a2633e49e63f0ec364c4b5dd27fd9d431f979e" Nov 02 17:55:54.335246 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.335162 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4945f033c3321035a66c43ef93a2633e49e63f0ec364c4b5dd27fd9d431f979e"} err="failed to get container status \"4945f033c3321035a66c43ef93a2633e49e63f0ec364c4b5dd27fd9d431f979e\": rpc error: code = NotFound desc = could not find container \"4945f033c3321035a66c43ef93a2633e49e63f0ec364c4b5dd27fd9d431f979e\": container with ID starting with 4945f033c3321035a66c43ef93a2633e49e63f0ec364c4b5dd27fd9d431f979e not found: ID does not exist" Nov 02 17:55:54.335246 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.335188 4860 scope.go:122] "RemoveContainer" containerID="99f2acd3cb7d3cdb010c6a333b6fa60263c0b73440eb502805ea70e952e02461" Nov 02 17:55:54.348649 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.348626 4860 scope.go:122] "RemoveContainer" containerID="99f2acd3cb7d3cdb010c6a333b6fa60263c0b73440eb502805ea70e952e02461" Nov 02 17:55:54.349194 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:55:54.349160 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99f2acd3cb7d3cdb010c6a333b6fa60263c0b73440eb502805ea70e952e02461\": container with ID starting with 99f2acd3cb7d3cdb010c6a333b6fa60263c0b73440eb502805ea70e952e02461 not found: ID does not exist" containerID="99f2acd3cb7d3cdb010c6a333b6fa60263c0b73440eb502805ea70e952e02461" Nov 02 17:55:54.349289 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.349204 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99f2acd3cb7d3cdb010c6a333b6fa60263c0b73440eb502805ea70e952e02461"} err="failed to get container status \"99f2acd3cb7d3cdb010c6a333b6fa60263c0b73440eb502805ea70e952e02461\": rpc error: code = NotFound desc = could not find container \"99f2acd3cb7d3cdb010c6a333b6fa60263c0b73440eb502805ea70e952e02461\": container with ID starting with 99f2acd3cb7d3cdb010c6a333b6fa60263c0b73440eb502805ea70e952e02461 not found: ID does not exist" Nov 02 17:55:54.408467 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.408433 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd"] Nov 02 17:55:54.412916 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.412887 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-znlsd"] Nov 02 17:55:54.442934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.442899 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b"] Nov 02 17:55:54.447528 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.447502 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-2k98b"] Nov 02 17:55:54.461438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.461414 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:55:54.563571 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.563506 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-workload-socket\") pod \"57a3d284-8cce-4c08-830a-605040cfb458\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " Nov 02 17:55:54.563714 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.563681 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-istio-data\") pod \"57a3d284-8cce-4c08-830a-605040cfb458\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " Nov 02 17:55:54.563791 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.563725 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57a3d284-8cce-4c08-830a-605040cfb458-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a3d284-8cce-4c08-830a-605040cfb458-istiod-ca-cert\") pod \"57a3d284-8cce-4c08-830a-605040cfb458\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " Nov 02 17:55:54.563791 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.563753 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-workload-socket" pod "57a3d284-8cce-4c08-830a-605040cfb458" (UID: "57a3d284-8cce-4c08-830a-605040cfb458"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:54.563950 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.563794 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a3d284-8cce-4c08-830a-605040cfb458-kube-api-access-kv7tx\" (UniqueName: \"kubernetes.io/projected/57a3d284-8cce-4c08-830a-605040cfb458-kube-api-access-kv7tx\") pod \"57a3d284-8cce-4c08-830a-605040cfb458\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " Nov 02 17:55:54.563950 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.563829 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-credential-socket\") pod \"57a3d284-8cce-4c08-830a-605040cfb458\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " Nov 02 17:55:54.563950 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.563895 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57a3d284-8cce-4c08-830a-605040cfb458-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a3d284-8cce-4c08-830a-605040cfb458-istio-podinfo\") pod \"57a3d284-8cce-4c08-830a-605040cfb458\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " Nov 02 17:55:54.564100 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.563956 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-istio-envoy\") pod \"57a3d284-8cce-4c08-830a-605040cfb458\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " Nov 02 17:55:54.564100 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.563984 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-istio-data" pod "57a3d284-8cce-4c08-830a-605040cfb458" (UID: "57a3d284-8cce-4c08-830a-605040cfb458"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:54.564100 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.564054 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a3d284-8cce-4c08-830a-605040cfb458-istio-token\" (UniqueName: \"kubernetes.io/projected/57a3d284-8cce-4c08-830a-605040cfb458-istio-token\") pod \"57a3d284-8cce-4c08-830a-605040cfb458\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " Nov 02 17:55:54.564248 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.564127 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-workload-certs\") pod \"57a3d284-8cce-4c08-830a-605040cfb458\" (UID: \"57a3d284-8cce-4c08-830a-605040cfb458\") " Nov 02 17:55:54.564248 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.564179 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-credential-socket" pod "57a3d284-8cce-4c08-830a-605040cfb458" (UID: "57a3d284-8cce-4c08-830a-605040cfb458"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:54.564248 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.564205 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a3d284-8cce-4c08-830a-605040cfb458-istiod-ca-cert" pod "57a3d284-8cce-4c08-830a-605040cfb458" (UID: "57a3d284-8cce-4c08-830a-605040cfb458"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:54.564528 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.564501 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-workload-certs" pod "57a3d284-8cce-4c08-830a-605040cfb458" (UID: "57a3d284-8cce-4c08-830a-605040cfb458"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:54.565187 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.565093 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.565187 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.565121 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.565187 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.565138 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a3d284-8cce-4c08-830a-605040cfb458-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.565187 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.565150 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.565187 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.565162 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.566236 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.566215 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a3d284-8cce-4c08-830a-605040cfb458-kube-api-access-kv7tx" pod "57a3d284-8cce-4c08-830a-605040cfb458" (UID: "57a3d284-8cce-4c08-830a-605040cfb458"). InnerVolumeSpecName "kube-api-access-kv7tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:54.566424 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.566403 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a3d284-8cce-4c08-830a-605040cfb458-istio-token" pod "57a3d284-8cce-4c08-830a-605040cfb458" (UID: "57a3d284-8cce-4c08-830a-605040cfb458"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:54.566593 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.566573 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-istio-envoy" pod "57a3d284-8cce-4c08-830a-605040cfb458" (UID: "57a3d284-8cce-4c08-830a-605040cfb458"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:54.566713 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.566689 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57a3d284-8cce-4c08-830a-605040cfb458-istio-podinfo" pod "57a3d284-8cce-4c08-830a-605040cfb458" (UID: "57a3d284-8cce-4c08-830a-605040cfb458"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:55:54.666404 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.666373 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv7tx\" (UniqueName: \"kubernetes.io/projected/57a3d284-8cce-4c08-830a-605040cfb458-kube-api-access-kv7tx\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.666404 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.666402 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a3d284-8cce-4c08-830a-605040cfb458-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.666404 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.666414 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a3d284-8cce-4c08-830a-605040cfb458-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:54.666649 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:54.666422 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57a3d284-8cce-4c08-830a-605040cfb458-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:55:55.320918 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:55.320884 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" Nov 02 17:55:55.320918 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:55.320901 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq" event={"ID":"57a3d284-8cce-4c08-830a-605040cfb458","Type":"ContainerDied","Data":"c768723eed2d9671919cabffd8d7c67420b6798c9fa51a06f7ae49ab1488fe71"} Nov 02 17:55:55.321463 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:55.320970 4860 scope.go:122] "RemoveContainer" containerID="189d517dfe3103337930a18e894cf7c35cba210420c6375c0c54fc085592bd96" Nov 02 17:55:55.387515 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:55.387484 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq"] Nov 02 17:55:55.399377 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:55.399342 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-kxzbq"] Nov 02 17:55:55.501731 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:55.501698 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b1a14b-2879-4835-acac-cfa13510175f" path="/var/lib/kubelet/pods/20b1a14b-2879-4835-acac-cfa13510175f/volumes" Nov 02 17:55:55.502392 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:55.502362 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49146668-6dbb-4df5-a1c1-b2c150a74979" path="/var/lib/kubelet/pods/49146668-6dbb-4df5-a1c1-b2c150a74979/volumes" Nov 02 17:55:55.502999 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:55:55.502982 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57a3d284-8cce-4c08-830a-605040cfb458" path="/var/lib/kubelet/pods/57a3d284-8cce-4c08-830a-605040cfb458/volumes" Nov 02 17:56:00.402696 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.402659 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd"] Nov 02 17:56:00.403679 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.403656 4860 cpu_manager.go:414] "containerMap: removing container" podUID="57a3d284-8cce-4c08-830a-605040cfb458" containerName="istio-proxy" Nov 02 17:56:00.403679 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.403678 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57a3d284-8cce-4c08-830a-605040cfb458" containerName="istio-proxy" Nov 02 17:56:00.403833 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.403714 4860 cpu_manager.go:414] "containerMap: removing container" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" Nov 02 17:56:00.403833 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.403720 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" Nov 02 17:56:00.403833 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.403743 4860 cpu_manager.go:414] "containerMap: removing container" podUID="49146668-6dbb-4df5-a1c1-b2c150a74979" containerName="istio-proxy" Nov 02 17:56:00.403833 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.403749 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49146668-6dbb-4df5-a1c1-b2c150a74979" containerName="istio-proxy" Nov 02 17:56:00.404028 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.403864 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="57a3d284-8cce-4c08-830a-605040cfb458" containerName="istio-proxy" Nov 02 17:56:00.404028 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.403885 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b1a14b-2879-4835-acac-cfa13510175f" containerName="istio-proxy" Nov 02 17:56:00.404028 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.403903 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="49146668-6dbb-4df5-a1c1-b2c150a74979" containerName="istio-proxy" Nov 02 17:56:00.420213 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.420180 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" Nov 02 17:56:00.429178 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.429134 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd"] Nov 02 17:56:00.436833 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.436805 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8a209f-serving-cert\"" Nov 02 17:56:00.556917 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.556880 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4332a04-bc02-4fce-925b-df037762c76f-maas-api-tls\") pod \"maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd\" (UID: \"d4332a04-bc02-4fce-925b-df037762c76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" Nov 02 17:56:00.557194 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.557153 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4332a04-bc02-4fce-925b-df037762c76f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd\" (UID: \"d4332a04-bc02-4fce-925b-df037762c76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" Nov 02 17:56:00.557316 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.557250 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t2nr\" (UniqueName: \"kubernetes.io/projected/d4332a04-bc02-4fce-925b-df037762c76f-kube-api-access-9t2nr\") pod \"maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd\" (UID: \"d4332a04-bc02-4fce-925b-df037762c76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" Nov 02 17:56:00.557411 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.557392 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4332a04-bc02-4fce-925b-df037762c76f-metrics-certs\") pod \"maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd\" (UID: \"d4332a04-bc02-4fce-925b-df037762c76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" Nov 02 17:56:00.659268 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.659170 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4332a04-bc02-4fce-925b-df037762c76f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd\" (UID: \"d4332a04-bc02-4fce-925b-df037762c76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" Nov 02 17:56:00.659268 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.659259 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t2nr\" (UniqueName: \"kubernetes.io/projected/d4332a04-bc02-4fce-925b-df037762c76f-kube-api-access-9t2nr\") pod \"maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd\" (UID: \"d4332a04-bc02-4fce-925b-df037762c76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" Nov 02 17:56:00.659497 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.659310 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4332a04-bc02-4fce-925b-df037762c76f-metrics-certs\") pod \"maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd\" (UID: \"d4332a04-bc02-4fce-925b-df037762c76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" Nov 02 17:56:00.659665 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.659601 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4332a04-bc02-4fce-925b-df037762c76f-maas-api-tls\") pod \"maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd\" (UID: \"d4332a04-bc02-4fce-925b-df037762c76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" Nov 02 17:56:00.659796 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:56:00.659733 4860 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-serving-cert: secret "maas-api-e2e-inf-b-8a209f-serving-cert" not found Nov 02 17:56:00.660058 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.660028 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4332a04-bc02-4fce-925b-df037762c76f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd\" (UID: \"d4332a04-bc02-4fce-925b-df037762c76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" Nov 02 17:56:00.660133 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:56:00.660073 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4332a04-bc02-4fce-925b-df037762c76f-maas-api-tls podName:d4332a04-bc02-4fce-925b-df037762c76f nodeName:}" failed. No retries permitted until 2026-11-02 17:56:01.160049352 +0000 UTC m=+1579.781409112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d4332a04-bc02-4fce-925b-df037762c76f-maas-api-tls") pod "maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" (UID: "d4332a04-bc02-4fce-925b-df037762c76f") : secret "maas-api-e2e-inf-b-8a209f-serving-cert" not found Nov 02 17:56:00.662597 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.662570 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4332a04-bc02-4fce-925b-df037762c76f-metrics-certs\") pod \"maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd\" (UID: \"d4332a04-bc02-4fce-925b-df037762c76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" Nov 02 17:56:00.684608 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:00.684581 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t2nr\" (UniqueName: \"kubernetes.io/projected/d4332a04-bc02-4fce-925b-df037762c76f-kube-api-access-9t2nr\") pod \"maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd\" (UID: \"d4332a04-bc02-4fce-925b-df037762c76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" Nov 02 17:56:01.168167 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:01.168128 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4332a04-bc02-4fce-925b-df037762c76f-maas-api-tls\") pod \"maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd\" (UID: \"d4332a04-bc02-4fce-925b-df037762c76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" Nov 02 17:56:01.170539 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:01.170516 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4332a04-bc02-4fce-925b-df037762c76f-maas-api-tls\") pod \"maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd\" (UID: \"d4332a04-bc02-4fce-925b-df037762c76f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" Nov 02 17:56:01.345387 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:01.345355 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" Nov 02 17:56:01.661654 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:01.661623 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd"] Nov 02 17:56:01.663515 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:56:01.663475 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4332a04_bc02_4fce_925b_df037762c76f.slice/crio-a75c1c11ab902d0efd3647dadc38d0d9026e45d508160a68748a2da612963d8b WatchSource:0}: Error finding container a75c1c11ab902d0efd3647dadc38d0d9026e45d508160a68748a2da612963d8b: Status 404 returned error can't find the container with id a75c1c11ab902d0efd3647dadc38d0d9026e45d508160a68748a2da612963d8b Nov 02 17:56:02.412159 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:02.412122 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" event={"ID":"d4332a04-bc02-4fce-925b-df037762c76f","Type":"ContainerStarted","Data":"e447a6570860e5b8a68b41bebf8f01847ac5d47119a361fe75ca14bdbdb62072"} Nov 02 17:56:02.412524 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:02.412188 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" event={"ID":"d4332a04-bc02-4fce-925b-df037762c76f","Type":"ContainerStarted","Data":"a75c1c11ab902d0efd3647dadc38d0d9026e45d508160a68748a2da612963d8b"} Nov 02 17:56:02.412524 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:02.412217 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" Nov 02 17:56:02.460662 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:02.460606 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" podStartSLOduration=2.084109177 podStartE2EDuration="2.460589614s" totalImagesPullingTime="376.480437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:02.455372052 +0000 UTC m=+1581.076731830" watchObservedRunningTime="2026-11-02 17:56:02.460589614 +0000 UTC m=+1581.081949389" Nov 02 17:56:08.425967 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:08.425936 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" Nov 02 17:56:08.905379 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:08.905335 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8"] Nov 02 17:56:08.905626 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:08.905593 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" podUID="2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" containerName="istio-proxy" containerID="cri-o://8a3e1c43680b4fb27a42a044d41ff690e46675736f8f14dcb228a6c437d7da1d" gracePeriod=30 Nov 02 17:56:08.914772 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:08.914741 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv"] Nov 02 17:56:08.915005 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:08.914978 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" podUID="cbdf6416-8c0d-40c7-978b-30a30f640b51" containerName="istio-proxy" containerID="cri-o://f22bd39d41120607cd0eb36ec9d99788b405a599c76082d9bdb627bd27c4b5e5" gracePeriod=30 Nov 02 17:56:12.011619 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:12.011585 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8"] Nov 02 17:56:12.017544 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:12.017519 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" Nov 02 17:56:12.043523 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:12.043136 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-bb6d2e-serving-cert\"" Nov 02 17:56:12.046915 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:12.046889 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8"] Nov 02 17:56:12.149757 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:12.149724 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63743df5-0f1d-4708-8c41-80b2c9344fc5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8\" (UID: \"63743df5-0f1d-4708-8c41-80b2c9344fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" Nov 02 17:56:12.150011 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:12.149959 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63743df5-0f1d-4708-8c41-80b2c9344fc5-maas-api-tls\") pod \"maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8\" (UID: \"63743df5-0f1d-4708-8c41-80b2c9344fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" Nov 02 17:56:12.150101 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:12.150085 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd6t9\" (UniqueName: \"kubernetes.io/projected/63743df5-0f1d-4708-8c41-80b2c9344fc5-kube-api-access-rd6t9\") pod \"maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8\" (UID: \"63743df5-0f1d-4708-8c41-80b2c9344fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" Nov 02 17:56:12.150169 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:12.150143 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63743df5-0f1d-4708-8c41-80b2c9344fc5-metrics-certs\") pod \"maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8\" (UID: \"63743df5-0f1d-4708-8c41-80b2c9344fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" Nov 02 17:56:12.258111 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:12.258061 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63743df5-0f1d-4708-8c41-80b2c9344fc5-maas-api-tls\") pod \"maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8\" (UID: \"63743df5-0f1d-4708-8c41-80b2c9344fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" Nov 02 17:56:12.258304 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:12.258203 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd6t9\" (UniqueName: \"kubernetes.io/projected/63743df5-0f1d-4708-8c41-80b2c9344fc5-kube-api-access-rd6t9\") pod \"maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8\" (UID: \"63743df5-0f1d-4708-8c41-80b2c9344fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" Nov 02 17:56:12.258304 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:12.258239 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63743df5-0f1d-4708-8c41-80b2c9344fc5-metrics-certs\") pod \"maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8\" (UID: \"63743df5-0f1d-4708-8c41-80b2c9344fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" Nov 02 17:56:12.258437 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:12.258372 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63743df5-0f1d-4708-8c41-80b2c9344fc5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8\" (UID: \"63743df5-0f1d-4708-8c41-80b2c9344fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" Nov 02 17:56:12.259356 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:12.259327 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63743df5-0f1d-4708-8c41-80b2c9344fc5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8\" (UID: \"63743df5-0f1d-4708-8c41-80b2c9344fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" Nov 02 17:56:12.267723 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:12.266919 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63743df5-0f1d-4708-8c41-80b2c9344fc5-maas-api-tls\") pod \"maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8\" (UID: \"63743df5-0f1d-4708-8c41-80b2c9344fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" Nov 02 17:56:12.275068 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:12.275016 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63743df5-0f1d-4708-8c41-80b2c9344fc5-metrics-certs\") pod \"maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8\" (UID: \"63743df5-0f1d-4708-8c41-80b2c9344fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" Nov 02 17:56:12.282212 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:12.282181 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd6t9\" (UniqueName: \"kubernetes.io/projected/63743df5-0f1d-4708-8c41-80b2c9344fc5-kube-api-access-rd6t9\") pod \"maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8\" (UID: \"63743df5-0f1d-4708-8c41-80b2c9344fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" Nov 02 17:56:12.344215 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:12.344172 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" Nov 02 17:56:12.659004 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:56:12.658967 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63743df5_0f1d_4708_8c41_80b2c9344fc5.slice/crio-cd182ca448e00ace50d5094b35bb7f8c81deff7fcc73b30c91ad36844055d00c WatchSource:0}: Error finding container cd182ca448e00ace50d5094b35bb7f8c81deff7fcc73b30c91ad36844055d00c: Status 404 returned error can't find the container with id cd182ca448e00ace50d5094b35bb7f8c81deff7fcc73b30c91ad36844055d00c Nov 02 17:56:12.660514 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:12.660487 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8"] Nov 02 17:56:13.507073 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:13.507025 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" event={"ID":"63743df5-0f1d-4708-8c41-80b2c9344fc5","Type":"ContainerStarted","Data":"8754e9920c9fe90cb663ed5ef8b60c9bfc7c08b2d87651fc2eb8896ca754adef"} Nov 02 17:56:13.507573 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:13.507085 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" event={"ID":"63743df5-0f1d-4708-8c41-80b2c9344fc5","Type":"ContainerStarted","Data":"cd182ca448e00ace50d5094b35bb7f8c81deff7fcc73b30c91ad36844055d00c"} Nov 02 17:56:13.507573 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:13.507174 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" Nov 02 17:56:13.557170 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:13.557116 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" podStartSLOduration=2.212481021 podStartE2EDuration="2.557101893s" totalImagesPullingTime="344.620872ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:13.555186518 +0000 UTC m=+1592.176546299" watchObservedRunningTime="2026-11-02 17:56:13.557101893 +0000 UTC m=+1592.178461701" Nov 02 17:56:13.591882 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:13.591811 4860 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:56:13.592055 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:13.591884 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" podUID="cbdf6416-8c0d-40c7-978b-30a30f640b51" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:56:13.648554 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:13.648507 4860 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:56:13.648752 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:13.648576 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" podUID="2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:56:14.248490 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.248461 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:56:14.289290 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.289265 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:56:14.399356 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.399313 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-credential-socket\") pod \"cbdf6416-8c0d-40c7-978b-30a30f640b51\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " Nov 02 17:56:14.399356 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.399354 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-data\") pod \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " Nov 02 17:56:14.399618 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.399385 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-workload-certs\") pod \"cbdf6416-8c0d-40c7-978b-30a30f640b51\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " Nov 02 17:56:14.399618 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.399415 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-envoy\") pod \"cbdf6416-8c0d-40c7-978b-30a30f640b51\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " Nov 02 17:56:14.399618 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.399490 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbdf6416-8c0d-40c7-978b-30a30f640b51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbdf6416-8c0d-40c7-978b-30a30f640b51-istiod-ca-cert\") pod \"cbdf6416-8c0d-40c7-978b-30a30f640b51\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " Nov 02 17:56:14.399618 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.399528 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-podinfo\") pod \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " Nov 02 17:56:14.399618 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.399568 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-podinfo\") pod \"cbdf6416-8c0d-40c7-978b-30a30f640b51\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " Nov 02 17:56:14.399956 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.399627 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-envoy\") pod \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " Nov 02 17:56:14.399956 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.399658 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-credential-socket" pod "cbdf6416-8c0d-40c7-978b-30a30f640b51" (UID: "cbdf6416-8c0d-40c7-978b-30a30f640b51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.399956 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.399663 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-data" pod "2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" (UID: "2a7e0db0-409a-4c80-b41c-7dfa0a531cd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.399956 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.399706 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-workload-certs\") pod \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " Nov 02 17:56:14.399956 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.399743 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-credential-socket\") pod \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " Nov 02 17:56:14.399956 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.399798 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istiod-ca-cert\") pod \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " Nov 02 17:56:14.399956 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.399829 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-workload-socket\") pod \"cbdf6416-8c0d-40c7-978b-30a30f640b51\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " Nov 02 17:56:14.399956 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.399911 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-kube-api-access-g8km6\" (UniqueName: \"kubernetes.io/projected/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-kube-api-access-g8km6\") pod \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " Nov 02 17:56:14.399956 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.399956 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-token\" (UniqueName: \"kubernetes.io/projected/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-token\") pod \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " Nov 02 17:56:14.400425 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.400122 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-data\") pod \"cbdf6416-8c0d-40c7-978b-30a30f640b51\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " Nov 02 17:56:14.400425 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.400205 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbdf6416-8c0d-40c7-978b-30a30f640b51-kube-api-access-dvdvq\" (UniqueName: \"kubernetes.io/projected/cbdf6416-8c0d-40c7-978b-30a30f640b51-kube-api-access-dvdvq\") pod \"cbdf6416-8c0d-40c7-978b-30a30f640b51\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " Nov 02 17:56:14.400425 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.400277 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-workload-socket\") pod \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\" (UID: \"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9\") " Nov 02 17:56:14.400425 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.400331 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-token\" (UniqueName: \"kubernetes.io/projected/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-token\") pod \"cbdf6416-8c0d-40c7-978b-30a30f640b51\" (UID: \"cbdf6416-8c0d-40c7-978b-30a30f640b51\") " Nov 02 17:56:14.400634 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.400497 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-credential-socket" pod "2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" (UID: "2a7e0db0-409a-4c80-b41c-7dfa0a531cd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.401132 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.401103 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbdf6416-8c0d-40c7-978b-30a30f640b51-istiod-ca-cert" pod "cbdf6416-8c0d-40c7-978b-30a30f640b51" (UID: "cbdf6416-8c0d-40c7-978b-30a30f640b51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:14.401277 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.401259 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.401982 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.401961 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.402091 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.401985 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbdf6416-8c0d-40c7-978b-30a30f640b51-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.402091 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.401998 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.402091 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.401442 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istiod-ca-cert" pod "2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" (UID: "2a7e0db0-409a-4c80-b41c-7dfa0a531cd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:14.402091 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.401463 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-workload-certs" pod "2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" (UID: "2a7e0db0-409a-4c80-b41c-7dfa0a531cd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.402091 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.401658 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-data" pod "cbdf6416-8c0d-40c7-978b-30a30f640b51" (UID: "cbdf6416-8c0d-40c7-978b-30a30f640b51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.402091 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.401696 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-workload-socket" pod "cbdf6416-8c0d-40c7-978b-30a30f640b51" (UID: "cbdf6416-8c0d-40c7-978b-30a30f640b51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.402419 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.402253 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-workload-certs" pod "cbdf6416-8c0d-40c7-978b-30a30f640b51" (UID: "cbdf6416-8c0d-40c7-978b-30a30f640b51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.402478 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.402444 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-workload-socket" pod "2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" (UID: "2a7e0db0-409a-4c80-b41c-7dfa0a531cd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.403136 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.403107 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-podinfo" pod "cbdf6416-8c0d-40c7-978b-30a30f640b51" (UID: "cbdf6416-8c0d-40c7-978b-30a30f640b51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:56:14.403617 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.403590 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-podinfo" pod "2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" (UID: "2a7e0db0-409a-4c80-b41c-7dfa0a531cd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:56:14.404656 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.404629 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdf6416-8c0d-40c7-978b-30a30f640b51-kube-api-access-dvdvq" pod "cbdf6416-8c0d-40c7-978b-30a30f640b51" (UID: "cbdf6416-8c0d-40c7-978b-30a30f640b51"). InnerVolumeSpecName "kube-api-access-dvdvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:14.404832 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.404807 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-token" pod "2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" (UID: "2a7e0db0-409a-4c80-b41c-7dfa0a531cd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:14.404952 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.404905 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-envoy" pod "2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" (UID: "2a7e0db0-409a-4c80-b41c-7dfa0a531cd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.405231 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.405208 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-kube-api-access-g8km6" pod "2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" (UID: "2a7e0db0-409a-4c80-b41c-7dfa0a531cd9"). InnerVolumeSpecName "kube-api-access-g8km6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:14.405843 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.405818 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-token" pod "cbdf6416-8c0d-40c7-978b-30a30f640b51" (UID: "cbdf6416-8c0d-40c7-978b-30a30f640b51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:14.406504 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.406480 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-envoy" pod "cbdf6416-8c0d-40c7-978b-30a30f640b51" (UID: "cbdf6416-8c0d-40c7-978b-30a30f640b51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:14.503596 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.503517 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.503596 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.503548 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.503596 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.503558 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.503596 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.503566 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.503596 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.503574 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.503596 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.503582 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.503596 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.503592 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8km6\" (UniqueName: \"kubernetes.io/projected/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-kube-api-access-g8km6\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.503596 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.503601 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.503994 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.503609 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.503994 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.503617 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvdvq\" (UniqueName: \"kubernetes.io/projected/cbdf6416-8c0d-40c7-978b-30a30f640b51-kube-api-access-dvdvq\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.503994 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.503627 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.503994 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.503636 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.503994 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.503645 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.503994 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.503652 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbdf6416-8c0d-40c7-978b-30a30f640b51-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:56:14.515897 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.515833 4860 generic.go:356] "Generic (PLEG): container finished" podID="2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" containerID="8a3e1c43680b4fb27a42a044d41ff690e46675736f8f14dcb228a6c437d7da1d" exitCode=0 Nov 02 17:56:14.515897 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.515875 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" event={"ID":"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9","Type":"ContainerDied","Data":"8a3e1c43680b4fb27a42a044d41ff690e46675736f8f14dcb228a6c437d7da1d"} Nov 02 17:56:14.515897 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.515896 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" Nov 02 17:56:14.516503 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.515933 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8" event={"ID":"2a7e0db0-409a-4c80-b41c-7dfa0a531cd9","Type":"ContainerDied","Data":"a73182a5032b420f6b5ac99345c68ff69acdfb484a2dd786296e055050636fac"} Nov 02 17:56:14.516503 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.515956 4860 scope.go:122] "RemoveContainer" containerID="8a3e1c43680b4fb27a42a044d41ff690e46675736f8f14dcb228a6c437d7da1d" Nov 02 17:56:14.517694 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.517666 4860 generic.go:356] "Generic (PLEG): container finished" podID="cbdf6416-8c0d-40c7-978b-30a30f640b51" containerID="f22bd39d41120607cd0eb36ec9d99788b405a599c76082d9bdb627bd27c4b5e5" exitCode=0 Nov 02 17:56:14.517835 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.517710 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" Nov 02 17:56:14.517835 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.517738 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" event={"ID":"cbdf6416-8c0d-40c7-978b-30a30f640b51","Type":"ContainerDied","Data":"f22bd39d41120607cd0eb36ec9d99788b405a599c76082d9bdb627bd27c4b5e5"} Nov 02 17:56:14.517835 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.517788 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv" event={"ID":"cbdf6416-8c0d-40c7-978b-30a30f640b51","Type":"ContainerDied","Data":"57dccc6818253903e74ccda61e53c9bc24ac2cf377114432b579f60d84dbb747"} Nov 02 17:56:14.528987 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.528969 4860 scope.go:122] "RemoveContainer" containerID="8a3e1c43680b4fb27a42a044d41ff690e46675736f8f14dcb228a6c437d7da1d" Nov 02 17:56:14.529266 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:56:14.529248 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a3e1c43680b4fb27a42a044d41ff690e46675736f8f14dcb228a6c437d7da1d\": container with ID starting with 8a3e1c43680b4fb27a42a044d41ff690e46675736f8f14dcb228a6c437d7da1d not found: ID does not exist" containerID="8a3e1c43680b4fb27a42a044d41ff690e46675736f8f14dcb228a6c437d7da1d" Nov 02 17:56:14.529310 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.529275 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a3e1c43680b4fb27a42a044d41ff690e46675736f8f14dcb228a6c437d7da1d"} err="failed to get container status \"8a3e1c43680b4fb27a42a044d41ff690e46675736f8f14dcb228a6c437d7da1d\": rpc error: code = NotFound desc = could not find container \"8a3e1c43680b4fb27a42a044d41ff690e46675736f8f14dcb228a6c437d7da1d\": container with ID starting with 8a3e1c43680b4fb27a42a044d41ff690e46675736f8f14dcb228a6c437d7da1d not found: ID does not exist" Nov 02 17:56:14.529310 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.529294 4860 scope.go:122] "RemoveContainer" containerID="f22bd39d41120607cd0eb36ec9d99788b405a599c76082d9bdb627bd27c4b5e5" Nov 02 17:56:14.540192 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.540164 4860 scope.go:122] "RemoveContainer" containerID="f22bd39d41120607cd0eb36ec9d99788b405a599c76082d9bdb627bd27c4b5e5" Nov 02 17:56:14.540488 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:56:14.540465 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f22bd39d41120607cd0eb36ec9d99788b405a599c76082d9bdb627bd27c4b5e5\": container with ID starting with f22bd39d41120607cd0eb36ec9d99788b405a599c76082d9bdb627bd27c4b5e5 not found: ID does not exist" containerID="f22bd39d41120607cd0eb36ec9d99788b405a599c76082d9bdb627bd27c4b5e5" Nov 02 17:56:14.540555 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.540499 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f22bd39d41120607cd0eb36ec9d99788b405a599c76082d9bdb627bd27c4b5e5"} err="failed to get container status \"f22bd39d41120607cd0eb36ec9d99788b405a599c76082d9bdb627bd27c4b5e5\": rpc error: code = NotFound desc = could not find container \"f22bd39d41120607cd0eb36ec9d99788b405a599c76082d9bdb627bd27c4b5e5\": container with ID starting with f22bd39d41120607cd0eb36ec9d99788b405a599c76082d9bdb627bd27c4b5e5 not found: ID does not exist" Nov 02 17:56:14.573314 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.573283 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv"] Nov 02 17:56:14.579621 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.579594 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-wqtsv"] Nov 02 17:56:14.614030 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.613998 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8"] Nov 02 17:56:14.618244 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:14.618217 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-7lgp8"] Nov 02 17:56:15.502125 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:15.502087 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" path="/var/lib/kubelet/pods/2a7e0db0-409a-4c80-b41c-7dfa0a531cd9/volumes" Nov 02 17:56:15.502945 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:15.502925 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbdf6416-8c0d-40c7-978b-30a30f640b51" path="/var/lib/kubelet/pods/cbdf6416-8c0d-40c7-978b-30a30f640b51/volumes" Nov 02 17:56:19.524394 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:56:19.524363 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" Nov 02 17:56:21.775785 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:56:21.775742 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:56:51.789210 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:56:51.789168 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:57:17.992890 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:17.992841 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq"] Nov 02 17:57:17.993564 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:17.993551 4860 cpu_manager.go:414] "containerMap: removing container" podUID="2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" containerName="istio-proxy" Nov 02 17:57:17.993564 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:17.993565 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" containerName="istio-proxy" Nov 02 17:57:17.993645 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:17.993588 4860 cpu_manager.go:414] "containerMap: removing container" podUID="cbdf6416-8c0d-40c7-978b-30a30f640b51" containerName="istio-proxy" Nov 02 17:57:17.993645 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:17.993593 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbdf6416-8c0d-40c7-978b-30a30f640b51" containerName="istio-proxy" Nov 02 17:57:17.993711 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:17.993689 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbdf6416-8c0d-40c7-978b-30a30f640b51" containerName="istio-proxy" Nov 02 17:57:17.993711 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:17.993704 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a7e0db0-409a-4c80-b41c-7dfa0a531cd9" containerName="istio-proxy" Nov 02 17:57:17.997713 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:17.997694 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" Nov 02 17:57:18.015448 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:18.015386 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b2fc9f-serving-cert\"" Nov 02 17:57:18.023159 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:18.023128 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq"] Nov 02 17:57:18.030811 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:18.030783 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e0682c-4519-452a-ab21-9781064a6260-metrics-certs\") pod \"maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq\" (UID: \"d7e0682c-4519-452a-ab21-9781064a6260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" Nov 02 17:57:18.031005 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:18.030843 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7e0682c-4519-452a-ab21-9781064a6260-maas-api-tls\") pod \"maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq\" (UID: \"d7e0682c-4519-452a-ab21-9781064a6260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" Nov 02 17:57:18.031005 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:18.030949 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzk6t\" (UniqueName: \"kubernetes.io/projected/d7e0682c-4519-452a-ab21-9781064a6260-kube-api-access-zzk6t\") pod \"maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq\" (UID: \"d7e0682c-4519-452a-ab21-9781064a6260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" Nov 02 17:57:18.031134 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:18.031070 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e0682c-4519-452a-ab21-9781064a6260-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq\" (UID: \"d7e0682c-4519-452a-ab21-9781064a6260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" Nov 02 17:57:18.133266 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:18.133210 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e0682c-4519-452a-ab21-9781064a6260-metrics-certs\") pod \"maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq\" (UID: \"d7e0682c-4519-452a-ab21-9781064a6260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" Nov 02 17:57:18.133470 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:18.133294 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7e0682c-4519-452a-ab21-9781064a6260-maas-api-tls\") pod \"maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq\" (UID: \"d7e0682c-4519-452a-ab21-9781064a6260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" Nov 02 17:57:18.133470 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:18.133383 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzk6t\" (UniqueName: \"kubernetes.io/projected/d7e0682c-4519-452a-ab21-9781064a6260-kube-api-access-zzk6t\") pod \"maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq\" (UID: \"d7e0682c-4519-452a-ab21-9781064a6260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" Nov 02 17:57:18.133470 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:57:18.133433 4860 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-serving-cert: secret "maas-api-e2e-inf-b-b2fc9f-serving-cert" not found Nov 02 17:57:18.133651 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:18.133494 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e0682c-4519-452a-ab21-9781064a6260-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq\" (UID: \"d7e0682c-4519-452a-ab21-9781064a6260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" Nov 02 17:57:18.133651 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:57:18.133608 4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7e0682c-4519-452a-ab21-9781064a6260-maas-api-tls podName:d7e0682c-4519-452a-ab21-9781064a6260 nodeName:}" failed. No retries permitted until 2026-11-02 17:57:18.633583698 +0000 UTC m=+1657.254943459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d7e0682c-4519-452a-ab21-9781064a6260-maas-api-tls") pod "maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" (UID: "d7e0682c-4519-452a-ab21-9781064a6260") : secret "maas-api-e2e-inf-b-b2fc9f-serving-cert" not found Nov 02 17:57:18.134285 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:18.134259 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e0682c-4519-452a-ab21-9781064a6260-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq\" (UID: \"d7e0682c-4519-452a-ab21-9781064a6260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" Nov 02 17:57:18.135944 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:18.135923 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e0682c-4519-452a-ab21-9781064a6260-metrics-certs\") pod \"maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq\" (UID: \"d7e0682c-4519-452a-ab21-9781064a6260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" Nov 02 17:57:18.157423 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:18.157392 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzk6t\" (UniqueName: \"kubernetes.io/projected/d7e0682c-4519-452a-ab21-9781064a6260-kube-api-access-zzk6t\") pod \"maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq\" (UID: \"d7e0682c-4519-452a-ab21-9781064a6260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" Nov 02 17:57:18.641129 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:18.641092 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7e0682c-4519-452a-ab21-9781064a6260-maas-api-tls\") pod \"maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq\" (UID: \"d7e0682c-4519-452a-ab21-9781064a6260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" Nov 02 17:57:18.643804 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:18.643774 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7e0682c-4519-452a-ab21-9781064a6260-maas-api-tls\") pod \"maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq\" (UID: \"d7e0682c-4519-452a-ab21-9781064a6260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" Nov 02 17:57:18.923715 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:18.923621 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" Nov 02 17:57:19.321406 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:19.321376 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq"] Nov 02 17:57:19.322749 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:57:19.322723 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e0682c_4519_452a_ab21_9781064a6260.slice/crio-4de9456a47e4b2fa65256a7c9f3c78d467b01a3e8aba4b28bd1512c036c68eaa WatchSource:0}: Error finding container 4de9456a47e4b2fa65256a7c9f3c78d467b01a3e8aba4b28bd1512c036c68eaa: Status 404 returned error can't find the container with id 4de9456a47e4b2fa65256a7c9f3c78d467b01a3e8aba4b28bd1512c036c68eaa Nov 02 17:57:19.368826 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:19.368539 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" event={"ID":"d7e0682c-4519-452a-ab21-9781064a6260","Type":"ContainerStarted","Data":"4de9456a47e4b2fa65256a7c9f3c78d467b01a3e8aba4b28bd1512c036c68eaa"} Nov 02 17:57:20.381509 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:20.381472 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" event={"ID":"d7e0682c-4519-452a-ab21-9781064a6260","Type":"ContainerStarted","Data":"45a94738647f7ba57a93a9ee4b6458dfacf4ec1ef9c72e9e3a5677d0fb802344"} Nov 02 17:57:20.381994 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:20.381716 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" Nov 02 17:57:20.429172 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:20.429107 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" podStartSLOduration=3.038995248 podStartE2EDuration="3.429088972s" totalImagesPullingTime="390.093724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:20.424838641 +0000 UTC m=+1659.046198418" watchObservedRunningTime="2026-11-02 17:57:20.429088972 +0000 UTC m=+1659.050448751" Nov 02 17:57:21.897997 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:57:21.897952 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:57:26.396166 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:26.396132 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" Nov 02 17:57:39.734006 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:39.733966 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs"] Nov 02 17:57:39.734901 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:39.734866 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" podUID="82e780ce-29d1-456e-985c-e3d9d61088eb" containerName="maas-api" containerID="cri-o://44b99dc5bfadfe9a7922c208ab963e3036713dddc3f3211c1a2f1be89991000e" gracePeriod=30 Nov 02 17:57:40.185158 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.185128 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" Nov 02 17:57:40.276874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.275691 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82e780ce-29d1-456e-985c-e3d9d61088eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e780ce-29d1-456e-985c-e3d9d61088eb-openshift-service-ca\") pod \"82e780ce-29d1-456e-985c-e3d9d61088eb\" (UID: \"82e780ce-29d1-456e-985c-e3d9d61088eb\") " Nov 02 17:57:40.276874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.275806 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e780ce-29d1-456e-985c-e3d9d61088eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82e780ce-29d1-456e-985c-e3d9d61088eb-maas-api-tls\") pod \"82e780ce-29d1-456e-985c-e3d9d61088eb\" (UID: \"82e780ce-29d1-456e-985c-e3d9d61088eb\") " Nov 02 17:57:40.276874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.276079 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e780ce-29d1-456e-985c-e3d9d61088eb-kube-api-access-9n9gn\" (UniqueName: \"kubernetes.io/projected/82e780ce-29d1-456e-985c-e3d9d61088eb-kube-api-access-9n9gn\") pod \"82e780ce-29d1-456e-985c-e3d9d61088eb\" (UID: \"82e780ce-29d1-456e-985c-e3d9d61088eb\") " Nov 02 17:57:40.276874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.276223 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e780ce-29d1-456e-985c-e3d9d61088eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e780ce-29d1-456e-985c-e3d9d61088eb-metrics-certs\") pod \"82e780ce-29d1-456e-985c-e3d9d61088eb\" (UID: \"82e780ce-29d1-456e-985c-e3d9d61088eb\") " Nov 02 17:57:40.276874 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.276631 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e780ce-29d1-456e-985c-e3d9d61088eb-openshift-service-ca" pod "82e780ce-29d1-456e-985c-e3d9d61088eb" (UID: "82e780ce-29d1-456e-985c-e3d9d61088eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:40.291933 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.277391 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e780ce-29d1-456e-985c-e3d9d61088eb-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:57:40.324073 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.322349 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e780ce-29d1-456e-985c-e3d9d61088eb-metrics-certs" pod "82e780ce-29d1-456e-985c-e3d9d61088eb" (UID: "82e780ce-29d1-456e-985c-e3d9d61088eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:57:40.360912 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.359609 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e780ce-29d1-456e-985c-e3d9d61088eb-kube-api-access-9n9gn" pod "82e780ce-29d1-456e-985c-e3d9d61088eb" (UID: "82e780ce-29d1-456e-985c-e3d9d61088eb"). InnerVolumeSpecName "kube-api-access-9n9gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:40.360912 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.359826 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e780ce-29d1-456e-985c-e3d9d61088eb-maas-api-tls" pod "82e780ce-29d1-456e-985c-e3d9d61088eb" (UID: "82e780ce-29d1-456e-985c-e3d9d61088eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:57:40.378752 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.378713 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n9gn\" (UniqueName: \"kubernetes.io/projected/82e780ce-29d1-456e-985c-e3d9d61088eb-kube-api-access-9n9gn\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:57:40.378905 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.378756 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e780ce-29d1-456e-985c-e3d9d61088eb-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:57:40.378905 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.378772 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82e780ce-29d1-456e-985c-e3d9d61088eb-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 17:57:40.552264 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.552177 4860 generic.go:356] "Generic (PLEG): container finished" podID="82e780ce-29d1-456e-985c-e3d9d61088eb" containerID="44b99dc5bfadfe9a7922c208ab963e3036713dddc3f3211c1a2f1be89991000e" exitCode=0 Nov 02 17:57:40.552264 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.552239 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" Nov 02 17:57:40.552449 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.552258 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" event={"ID":"82e780ce-29d1-456e-985c-e3d9d61088eb","Type":"ContainerDied","Data":"44b99dc5bfadfe9a7922c208ab963e3036713dddc3f3211c1a2f1be89991000e"} Nov 02 17:57:40.552449 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.552315 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs" event={"ID":"82e780ce-29d1-456e-985c-e3d9d61088eb","Type":"ContainerDied","Data":"a8a991d56f7e8899eb233cc56fe4e8c42608ef64c1b15b0a9a4114e7314fbca1"} Nov 02 17:57:40.552449 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.552340 4860 scope.go:122] "RemoveContainer" containerID="44b99dc5bfadfe9a7922c208ab963e3036713dddc3f3211c1a2f1be89991000e" Nov 02 17:57:40.569078 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.569047 4860 scope.go:122] "RemoveContainer" containerID="44b99dc5bfadfe9a7922c208ab963e3036713dddc3f3211c1a2f1be89991000e" Nov 02 17:57:40.569615 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:57:40.569584 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44b99dc5bfadfe9a7922c208ab963e3036713dddc3f3211c1a2f1be89991000e\": container with ID starting with 44b99dc5bfadfe9a7922c208ab963e3036713dddc3f3211c1a2f1be89991000e not found: ID does not exist" containerID="44b99dc5bfadfe9a7922c208ab963e3036713dddc3f3211c1a2f1be89991000e" Nov 02 17:57:40.569842 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.569810 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44b99dc5bfadfe9a7922c208ab963e3036713dddc3f3211c1a2f1be89991000e"} err="failed to get container status \"44b99dc5bfadfe9a7922c208ab963e3036713dddc3f3211c1a2f1be89991000e\": rpc error: code = NotFound desc = could not find container \"44b99dc5bfadfe9a7922c208ab963e3036713dddc3f3211c1a2f1be89991000e\": container with ID starting with 44b99dc5bfadfe9a7922c208ab963e3036713dddc3f3211c1a2f1be89991000e not found: ID does not exist" Nov 02 17:57:40.637754 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.637716 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs"] Nov 02 17:57:40.647716 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:40.647689 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-227833-6658b99d67-2pgfs"] Nov 02 17:57:41.501122 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:41.501090 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e780ce-29d1-456e-985c-e3d9d61088eb" path="/var/lib/kubelet/pods/82e780ce-29d1-456e-985c-e3d9d61088eb/volumes" Nov 02 17:57:44.183415 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.183380 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc"] Nov 02 17:57:44.184603 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.184576 4860 cpu_manager.go:414] "containerMap: removing container" podUID="82e780ce-29d1-456e-985c-e3d9d61088eb" containerName="maas-api" Nov 02 17:57:44.184603 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.184603 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e780ce-29d1-456e-985c-e3d9d61088eb" containerName="maas-api" Nov 02 17:57:44.184779 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.184766 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e780ce-29d1-456e-985c-e3d9d61088eb" containerName="maas-api" Nov 02 17:57:44.190529 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.190501 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.217023 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.216989 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc"] Nov 02 17:57:44.222516 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.222481 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-063f48-openshift-default-dockercfg-n95mk\"" Nov 02 17:57:44.350415 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.350378 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-workload-socket\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.350415 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.350419 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-token\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.350622 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.350530 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-data\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.350622 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.350577 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-podinfo\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.350690 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.350640 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-envoy\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.350726 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.350701 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istiod-ca-cert\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.350761 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.350736 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-credential-socket\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.350830 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.350816 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2sgs\" (UniqueName: \"kubernetes.io/projected/7bba4e6a-f4d0-4840-8e6d-f229f330a336-kube-api-access-r2sgs\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.350940 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.350928 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-workload-certs\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.452193 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.452111 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-workload-certs\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.452193 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.452185 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-workload-socket\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.452418 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.452208 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-token\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.452418 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.452251 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-data\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.452418 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.452282 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-podinfo\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.452418 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.452327 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-envoy\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.452418 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.452368 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istiod-ca-cert\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.452707 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.452426 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-credential-socket\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.452707 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.452574 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2sgs\" (UniqueName: \"kubernetes.io/projected/7bba4e6a-f4d0-4840-8e6d-f229f330a336-kube-api-access-r2sgs\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.452707 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.452688 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-workload-certs\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.452909 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.452728 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-workload-socket\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.452909 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.452799 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-data\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.453056 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.453037 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-credential-socket\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.453282 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.453263 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istiod-ca-cert\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.454858 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.454831 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-envoy\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.454921 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.454904 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-podinfo\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.477294 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.477271 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2sgs\" (UniqueName: \"kubernetes.io/projected/7bba4e6a-f4d0-4840-8e6d-f229f330a336-kube-api-access-r2sgs\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.482994 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.482966 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-token\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.505925 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.505900 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:44.838740 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:57:44.838708 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bba4e6a_f4d0_4840_8e6d_f229f330a336.slice/crio-1b1bc243a9384b9b8bbe71858b9b43b43a29db30ef1596b8f38ad54dcf5b019d WatchSource:0}: Error finding container 1b1bc243a9384b9b8bbe71858b9b43b43a29db30ef1596b8f38ad54dcf5b019d: Status 404 returned error can't find the container with id 1b1bc243a9384b9b8bbe71858b9b43b43a29db30ef1596b8f38ad54dcf5b019d Nov 02 17:57:44.839621 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.839595 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc"] Nov 02 17:57:44.841558 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.841523 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:57:44.841671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.841616 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:57:44.841671 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:44.841663 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:57:45.600398 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:45.600359 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" event={"ID":"7bba4e6a-f4d0-4840-8e6d-f229f330a336","Type":"ContainerStarted","Data":"36099368465b567ff1a182394d85986401c2a4e3953fb4eb9ee9f258b4324815"} Nov 02 17:57:45.600805 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:45.600422 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" event={"ID":"7bba4e6a-f4d0-4840-8e6d-f229f330a336","Type":"ContainerStarted","Data":"1b1bc243a9384b9b8bbe71858b9b43b43a29db30ef1596b8f38ad54dcf5b019d"} Nov 02 17:57:45.684343 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:45.684285 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" podStartSLOduration=1.6842711910000001 podStartE2EDuration="1.684271191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:45.675410747 +0000 UTC m=+1684.296770541" watchObservedRunningTime="2026-11-02 17:57:45.684271191 +0000 UTC m=+1684.305630967" Nov 02 17:57:46.506402 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:46.506367 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:46.508253 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:46.508229 4860 patch_prober.go:29] interesting pod/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 17:57:46.508426 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:46.508277 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" podUID="7bba4e6a-f4d0-4840-8e6d-f229f330a336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 17:57:47.510273 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:47.510232 4860 patch_prober.go:29] interesting pod/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 17:57:47.510753 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:47.510312 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" podUID="7bba4e6a-f4d0-4840-8e6d-f229f330a336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 17:57:48.523872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:48.522948 4860 patch_prober.go:29] interesting pod/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 17:57:48.523872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:48.523019 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" podUID="7bba4e6a-f4d0-4840-8e6d-f229f330a336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 17:57:49.506949 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:49.506905 4860 patch_prober.go:29] interesting pod/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 17:57:49.507208 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:49.506972 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" podUID="7bba4e6a-f4d0-4840-8e6d-f229f330a336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 17:57:50.507064 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:50.507012 4860 patch_prober.go:29] interesting pod/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 17:57:50.507564 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:50.507103 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" podUID="7bba4e6a-f4d0-4840-8e6d-f229f330a336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 17:57:51.518168 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:51.518129 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:51.518727 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:51.518705 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:51.519644 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:51.519623 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 17:57:51.777492 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:57:51.777409 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:57:52.504661 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:52.504626 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb"] Nov 02 17:57:52.511176 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:52.511150 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" Nov 02 17:57:52.535385 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:52.535348 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb"] Nov 02 17:57:52.536702 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:52.536417 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d5b905-serving-cert\"" Nov 02 17:57:52.669426 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:52.669394 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqnz6\" (UniqueName: \"kubernetes.io/projected/6277b932-8a69-4947-afe2-0946a7c6f177-kube-api-access-hqnz6\") pod \"maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb\" (UID: \"6277b932-8a69-4947-afe2-0946a7c6f177\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" Nov 02 17:57:52.669602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:52.669535 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6277b932-8a69-4947-afe2-0946a7c6f177-metrics-certs\") pod \"maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb\" (UID: \"6277b932-8a69-4947-afe2-0946a7c6f177\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" Nov 02 17:57:52.669651 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:52.669609 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6277b932-8a69-4947-afe2-0946a7c6f177-maas-api-tls\") pod \"maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb\" (UID: \"6277b932-8a69-4947-afe2-0946a7c6f177\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" Nov 02 17:57:52.669713 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:52.669700 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6277b932-8a69-4947-afe2-0946a7c6f177-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb\" (UID: \"6277b932-8a69-4947-afe2-0946a7c6f177\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" Nov 02 17:57:52.772219 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:52.771868 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqnz6\" (UniqueName: \"kubernetes.io/projected/6277b932-8a69-4947-afe2-0946a7c6f177-kube-api-access-hqnz6\") pod \"maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb\" (UID: \"6277b932-8a69-4947-afe2-0946a7c6f177\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" Nov 02 17:57:52.772219 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:52.772053 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6277b932-8a69-4947-afe2-0946a7c6f177-metrics-certs\") pod \"maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb\" (UID: \"6277b932-8a69-4947-afe2-0946a7c6f177\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" Nov 02 17:57:52.772219 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:52.772139 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6277b932-8a69-4947-afe2-0946a7c6f177-maas-api-tls\") pod \"maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb\" (UID: \"6277b932-8a69-4947-afe2-0946a7c6f177\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" Nov 02 17:57:52.772693 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:52.772257 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6277b932-8a69-4947-afe2-0946a7c6f177-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb\" (UID: \"6277b932-8a69-4947-afe2-0946a7c6f177\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" Nov 02 17:57:52.773164 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:52.773135 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6277b932-8a69-4947-afe2-0946a7c6f177-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb\" (UID: \"6277b932-8a69-4947-afe2-0946a7c6f177\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" Nov 02 17:57:52.776805 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:52.776262 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6277b932-8a69-4947-afe2-0946a7c6f177-maas-api-tls\") pod \"maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb\" (UID: \"6277b932-8a69-4947-afe2-0946a7c6f177\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" Nov 02 17:57:52.776805 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:52.776277 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6277b932-8a69-4947-afe2-0946a7c6f177-metrics-certs\") pod \"maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb\" (UID: \"6277b932-8a69-4947-afe2-0946a7c6f177\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" Nov 02 17:57:52.798490 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:52.798448 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqnz6\" (UniqueName: \"kubernetes.io/projected/6277b932-8a69-4947-afe2-0946a7c6f177-kube-api-access-hqnz6\") pod \"maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb\" (UID: \"6277b932-8a69-4947-afe2-0946a7c6f177\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" Nov 02 17:57:52.843053 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:52.843002 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" Nov 02 17:57:53.168153 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:57:53.168120 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6277b932_8a69_4947_afe2_0946a7c6f177.slice/crio-c93d1acfd8b5ed2b4d10971e1bc4f2dd15e2713d846542a79a2b539de24bb577 WatchSource:0}: Error finding container c93d1acfd8b5ed2b4d10971e1bc4f2dd15e2713d846542a79a2b539de24bb577: Status 404 returned error can't find the container with id c93d1acfd8b5ed2b4d10971e1bc4f2dd15e2713d846542a79a2b539de24bb577 Nov 02 17:57:53.168278 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:53.168165 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb"] Nov 02 17:57:53.743802 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:53.743715 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" event={"ID":"6277b932-8a69-4947-afe2-0946a7c6f177","Type":"ContainerStarted","Data":"fec8d6a1326506e9d2303346e455516e0ad98a3b27c19b55905f9a6c2f591199"} Nov 02 17:57:53.743802 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:53.743775 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" event={"ID":"6277b932-8a69-4947-afe2-0946a7c6f177","Type":"ContainerStarted","Data":"c93d1acfd8b5ed2b4d10971e1bc4f2dd15e2713d846542a79a2b539de24bb577"} Nov 02 17:57:53.744754 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:53.744735 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" Nov 02 17:57:53.796492 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:57:53.796437 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" podStartSLOduration=1.478233109 podStartE2EDuration="1.79642268s" totalImagesPullingTime="318.189571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:53.792430795 +0000 UTC m=+1692.413790572" watchObservedRunningTime="2026-11-02 17:57:53.79642268 +0000 UTC m=+1692.417782457" Nov 02 17:58:00.777055 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:00.777022 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" Nov 02 17:58:15.331105 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.331068 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h"] Nov 02 17:58:15.336264 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.336242 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:15.357448 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.357415 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h"] Nov 02 17:58:15.362565 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.362540 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a5c1da\"/\"default-dockercfg-54xp9\"" Nov 02 17:58:15.362741 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.362540 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a5c1da\"/\"openshift-service-ca.crt\"" Nov 02 17:58:15.365144 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.365123 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a5c1da\"/\"kube-root-ca.crt\"" Nov 02 17:58:15.365263 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.365242 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a5c1da\"/\"test-model-a5c1da-kserve-self-signed-certs\"" Nov 02 17:58:15.471736 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.471694 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-tmp-dir\") pod \"test-model-a5c1da-kserve-68ffbb7dc7-q9t2h\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:15.471929 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.471754 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-home\") pod \"test-model-a5c1da-kserve-68ffbb7dc7-q9t2h\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:15.472035 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.472013 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5bgp\" (UniqueName: \"kubernetes.io/projected/48350106-29a0-4a22-a995-203997f1d909-kube-api-access-g5bgp\") pod \"test-model-a5c1da-kserve-68ffbb7dc7-q9t2h\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:15.472087 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.472071 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48350106-29a0-4a22-a995-203997f1d909-tls-certs\") pod \"test-model-a5c1da-kserve-68ffbb7dc7-q9t2h\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:15.472199 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.472185 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-dshm\") pod \"test-model-a5c1da-kserve-68ffbb7dc7-q9t2h\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:15.472257 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.472243 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-model-cache\") pod \"test-model-a5c1da-kserve-68ffbb7dc7-q9t2h\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:15.583924 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.582868 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-model-cache\") pod \"test-model-a5c1da-kserve-68ffbb7dc7-q9t2h\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:15.583924 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.583033 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-tmp-dir\") pod \"test-model-a5c1da-kserve-68ffbb7dc7-q9t2h\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:15.583924 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.583075 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-home\") pod \"test-model-a5c1da-kserve-68ffbb7dc7-q9t2h\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:15.583924 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.583276 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5bgp\" (UniqueName: \"kubernetes.io/projected/48350106-29a0-4a22-a995-203997f1d909-kube-api-access-g5bgp\") pod \"test-model-a5c1da-kserve-68ffbb7dc7-q9t2h\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:15.583924 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.583361 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48350106-29a0-4a22-a995-203997f1d909-tls-certs\") pod \"test-model-a5c1da-kserve-68ffbb7dc7-q9t2h\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:15.583924 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.583473 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-dshm\") pod \"test-model-a5c1da-kserve-68ffbb7dc7-q9t2h\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:15.592175 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.589586 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-model-cache\") pod \"test-model-a5c1da-kserve-68ffbb7dc7-q9t2h\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:15.592175 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.589899 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-tmp-dir\") pod \"test-model-a5c1da-kserve-68ffbb7dc7-q9t2h\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:15.592175 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.590155 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-home\") pod \"test-model-a5c1da-kserve-68ffbb7dc7-q9t2h\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:15.594098 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.594065 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-dshm\") pod \"test-model-a5c1da-kserve-68ffbb7dc7-q9t2h\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:15.612917 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.610911 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48350106-29a0-4a22-a995-203997f1d909-tls-certs\") pod \"test-model-a5c1da-kserve-68ffbb7dc7-q9t2h\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:15.619275 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.619244 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5bgp\" (UniqueName: \"kubernetes.io/projected/48350106-29a0-4a22-a995-203997f1d909-kube-api-access-g5bgp\") pod \"test-model-a5c1da-kserve-68ffbb7dc7-q9t2h\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:15.657333 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:15.657296 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:16.022909 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:16.022874 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h"] Nov 02 17:58:17.035289 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:17.035248 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" event={"ID":"48350106-29a0-4a22-a995-203997f1d909","Type":"ContainerStarted","Data":"b98a4fce807a65fb27d604360bab90cfc1da48ebc82d039421e83be7228991e8"} Nov 02 17:58:17.035774 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:17.035301 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" event={"ID":"48350106-29a0-4a22-a995-203997f1d909","Type":"ContainerStarted","Data":"a34ddaa22c7195e13674daf34ef85feab03f3e8e0cc370809e1762f1652413dd"} Nov 02 17:58:17.078435 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:17.078375 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" podStartSLOduration=2.078360524 podStartE2EDuration="2.078360524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:17.077336729 +0000 UTC m=+1715.698696505" watchObservedRunningTime="2026-11-02 17:58:17.078360524 +0000 UTC m=+1715.699720291" Nov 02 17:58:17.659351 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:17.659318 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:21.773703 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:58:21.773664 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:58:25.659092 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:25.659055 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:25.672383 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:25.672359 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:26.130462 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:26.130429 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 17:58:28.305129 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:28.305099 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn"] Nov 02 17:58:28.311634 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:28.311613 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" Nov 02 17:58:28.330914 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:28.330881 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn"] Nov 02 17:58:28.331779 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:28.331755 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-96768b-serving-cert\"" Nov 02 17:58:28.381221 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:28.381185 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn44b\" (UniqueName: \"kubernetes.io/projected/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-kube-api-access-sn44b\") pod \"maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn\" (UID: \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" Nov 02 17:58:28.381385 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:28.381235 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-openshift-service-ca\") pod \"maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn\" (UID: \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" Nov 02 17:58:28.381453 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:28.381426 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-maas-api-tls\") pod \"maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn\" (UID: \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" Nov 02 17:58:28.381640 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:28.381611 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-metrics-certs\") pod \"maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn\" (UID: \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" Nov 02 17:58:28.488448 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:28.488398 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn44b\" (UniqueName: \"kubernetes.io/projected/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-kube-api-access-sn44b\") pod \"maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn\" (UID: \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" Nov 02 17:58:28.488756 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:28.488725 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-openshift-service-ca\") pod \"maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn\" (UID: \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" Nov 02 17:58:28.489045 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:28.488983 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-maas-api-tls\") pod \"maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn\" (UID: \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" Nov 02 17:58:28.489142 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:28.489123 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-metrics-certs\") pod \"maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn\" (UID: \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" Nov 02 17:58:28.489657 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:28.489630 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-openshift-service-ca\") pod \"maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn\" (UID: \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" Nov 02 17:58:28.492996 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:28.492966 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-metrics-certs\") pod \"maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn\" (UID: \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" Nov 02 17:58:28.496007 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:28.495910 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-maas-api-tls\") pod \"maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn\" (UID: \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" Nov 02 17:58:28.512894 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:28.512834 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn44b\" (UniqueName: \"kubernetes.io/projected/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-kube-api-access-sn44b\") pod \"maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn\" (UID: \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" Nov 02 17:58:28.631901 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:28.631790 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" Nov 02 17:58:28.948689 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:58:28.948653 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9aa9e4_1f8b_4512_ae25_51b759bd1c91.slice/crio-38a9f74b44831d367635be12e2f05135edfaf61c5e08c6f5bb12dc09faed940a WatchSource:0}: Error finding container 38a9f74b44831d367635be12e2f05135edfaf61c5e08c6f5bb12dc09faed940a: Status 404 returned error can't find the container with id 38a9f74b44831d367635be12e2f05135edfaf61c5e08c6f5bb12dc09faed940a Nov 02 17:58:28.949286 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:28.949213 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn"] Nov 02 17:58:28.950243 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:28.950223 4860 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:58:29.151243 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:29.151192 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" event={"ID":"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91","Type":"ContainerStarted","Data":"38a9f74b44831d367635be12e2f05135edfaf61c5e08c6f5bb12dc09faed940a"} Nov 02 17:58:30.162989 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:30.162926 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" event={"ID":"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91","Type":"ContainerStarted","Data":"8d884b62d3605e5831151ba07bfd94384cdeeead146e4db5f5c56b05777993bd"} Nov 02 17:58:30.163496 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:30.163041 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" Nov 02 17:58:30.210491 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:30.210433 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" podStartSLOduration=1.914095177 podStartE2EDuration="2.210419023s" totalImagesPullingTime="296.323846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:30.20678471 +0000 UTC m=+1728.828144483" watchObservedRunningTime="2026-11-02 17:58:30.210419023 +0000 UTC m=+1728.831778799" Nov 02 17:58:36.177759 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:36.177729 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" Nov 02 17:58:41.612939 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:41.612899 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk"] Nov 02 17:58:41.623152 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:41.623124 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk"] Nov 02 17:58:41.626608 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:41.626558 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" Nov 02 17:58:41.642115 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:41.642090 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-872354-serving-cert\"" Nov 02 17:58:41.736628 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:41.736591 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66504ddb-d869-4549-b870-61b42e58e871-openshift-service-ca\") pod \"maas-api-e2e-inf-a-872354-57d7f6556c-swnrk\" (UID: \"66504ddb-d869-4549-b870-61b42e58e871\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" Nov 02 17:58:41.736809 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:41.736733 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66504ddb-d869-4549-b870-61b42e58e871-maas-api-tls\") pod \"maas-api-e2e-inf-a-872354-57d7f6556c-swnrk\" (UID: \"66504ddb-d869-4549-b870-61b42e58e871\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" Nov 02 17:58:41.736809 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:41.736786 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66504ddb-d869-4549-b870-61b42e58e871-metrics-certs\") pod \"maas-api-e2e-inf-a-872354-57d7f6556c-swnrk\" (UID: \"66504ddb-d869-4549-b870-61b42e58e871\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" Nov 02 17:58:41.737000 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:41.736981 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69d4\" (UniqueName: \"kubernetes.io/projected/66504ddb-d869-4549-b870-61b42e58e871-kube-api-access-k69d4\") pod \"maas-api-e2e-inf-a-872354-57d7f6556c-swnrk\" (UID: \"66504ddb-d869-4549-b870-61b42e58e871\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" Nov 02 17:58:41.838976 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:41.838931 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69d4\" (UniqueName: \"kubernetes.io/projected/66504ddb-d869-4549-b870-61b42e58e871-kube-api-access-k69d4\") pod \"maas-api-e2e-inf-a-872354-57d7f6556c-swnrk\" (UID: \"66504ddb-d869-4549-b870-61b42e58e871\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" Nov 02 17:58:41.839214 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:41.839183 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66504ddb-d869-4549-b870-61b42e58e871-openshift-service-ca\") pod \"maas-api-e2e-inf-a-872354-57d7f6556c-swnrk\" (UID: \"66504ddb-d869-4549-b870-61b42e58e871\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" Nov 02 17:58:41.839486 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:41.839463 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66504ddb-d869-4549-b870-61b42e58e871-maas-api-tls\") pod \"maas-api-e2e-inf-a-872354-57d7f6556c-swnrk\" (UID: \"66504ddb-d869-4549-b870-61b42e58e871\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" Nov 02 17:58:41.839586 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:41.839510 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66504ddb-d869-4549-b870-61b42e58e871-metrics-certs\") pod \"maas-api-e2e-inf-a-872354-57d7f6556c-swnrk\" (UID: \"66504ddb-d869-4549-b870-61b42e58e871\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" Nov 02 17:58:41.840011 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:41.839987 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66504ddb-d869-4549-b870-61b42e58e871-openshift-service-ca\") pod \"maas-api-e2e-inf-a-872354-57d7f6556c-swnrk\" (UID: \"66504ddb-d869-4549-b870-61b42e58e871\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" Nov 02 17:58:41.842298 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:41.842277 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66504ddb-d869-4549-b870-61b42e58e871-maas-api-tls\") pod \"maas-api-e2e-inf-a-872354-57d7f6556c-swnrk\" (UID: \"66504ddb-d869-4549-b870-61b42e58e871\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" Nov 02 17:58:41.842405 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:41.842387 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66504ddb-d869-4549-b870-61b42e58e871-metrics-certs\") pod \"maas-api-e2e-inf-a-872354-57d7f6556c-swnrk\" (UID: \"66504ddb-d869-4549-b870-61b42e58e871\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" Nov 02 17:58:41.866617 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:41.866545 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69d4\" (UniqueName: \"kubernetes.io/projected/66504ddb-d869-4549-b870-61b42e58e871-kube-api-access-k69d4\") pod \"maas-api-e2e-inf-a-872354-57d7f6556c-swnrk\" (UID: \"66504ddb-d869-4549-b870-61b42e58e871\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" Nov 02 17:58:41.948937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:41.948890 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" Nov 02 17:58:42.273671 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:58:42.273641 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66504ddb_d869_4549_b870_61b42e58e871.slice/crio-154a26262210e0273e8569c2fb88928153bccbc59a752b0533381471bc778e89 WatchSource:0}: Error finding container 154a26262210e0273e8569c2fb88928153bccbc59a752b0533381471bc778e89: Status 404 returned error can't find the container with id 154a26262210e0273e8569c2fb88928153bccbc59a752b0533381471bc778e89 Nov 02 17:58:42.274644 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:42.274622 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk"] Nov 02 17:58:43.282086 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:43.282049 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" event={"ID":"66504ddb-d869-4549-b870-61b42e58e871","Type":"ContainerStarted","Data":"060ba0b0295fab8763442d2f97fd2a58e32cf4e65d63f7f9b59611601e1cc01c"} Nov 02 17:58:43.282512 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:43.282108 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" event={"ID":"66504ddb-d869-4549-b870-61b42e58e871","Type":"ContainerStarted","Data":"154a26262210e0273e8569c2fb88928153bccbc59a752b0533381471bc778e89"} Nov 02 17:58:43.282512 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:43.282139 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" Nov 02 17:58:43.328566 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:43.328513 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" podStartSLOduration=1.92485728 podStartE2EDuration="2.32849933s" totalImagesPullingTime="403.64205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:43.324477343 +0000 UTC m=+1741.945837131" watchObservedRunningTime="2026-11-02 17:58:43.32849933 +0000 UTC m=+1741.949859105" Nov 02 17:58:44.701335 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.701301 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf"] Nov 02 17:58:44.706807 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.706781 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:44.725154 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.724080 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b2fc9f\"/\"openshift-service-ca.crt\"" Nov 02 17:58:44.725154 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.724218 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b2fc9f\"/\"kube-root-ca.crt\"" Nov 02 17:58:44.733125 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.733096 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf"] Nov 02 17:58:44.740570 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.740547 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b2fc9f\"/\"test-model-b2fc9f-kserve-self-signed-certs\"" Nov 02 17:58:44.740717 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.740670 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b2fc9f\"/\"default-dockercfg-xhg6q\"" Nov 02 17:58:44.790392 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.790360 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-home\") pod \"test-model-b2fc9f-kserve-6f9cb5f967-8cbzf\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:44.790567 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.790404 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-tls-certs\") pod \"test-model-b2fc9f-kserve-6f9cb5f967-8cbzf\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:44.790567 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.790521 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-model-cache\") pod \"test-model-b2fc9f-kserve-6f9cb5f967-8cbzf\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:44.790567 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.790543 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-dshm\") pod \"test-model-b2fc9f-kserve-6f9cb5f967-8cbzf\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:44.790567 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.790563 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4w8g\" (UniqueName: \"kubernetes.io/projected/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-kube-api-access-b4w8g\") pod \"test-model-b2fc9f-kserve-6f9cb5f967-8cbzf\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:44.790922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.790889 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-tmp-dir\") pod \"test-model-b2fc9f-kserve-6f9cb5f967-8cbzf\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:44.892419 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.892385 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-tmp-dir\") pod \"test-model-b2fc9f-kserve-6f9cb5f967-8cbzf\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:44.892621 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.892485 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-home\") pod \"test-model-b2fc9f-kserve-6f9cb5f967-8cbzf\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:44.892621 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.892524 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-tls-certs\") pod \"test-model-b2fc9f-kserve-6f9cb5f967-8cbzf\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:44.892621 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.892581 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-model-cache\") pod \"test-model-b2fc9f-kserve-6f9cb5f967-8cbzf\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:44.892621 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.892601 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-dshm\") pod \"test-model-b2fc9f-kserve-6f9cb5f967-8cbzf\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:44.892934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.892627 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4w8g\" (UniqueName: \"kubernetes.io/projected/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-kube-api-access-b4w8g\") pod \"test-model-b2fc9f-kserve-6f9cb5f967-8cbzf\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:44.892934 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.892879 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-tmp-dir\") pod \"test-model-b2fc9f-kserve-6f9cb5f967-8cbzf\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:44.893048 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.893033 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-home\") pod \"test-model-b2fc9f-kserve-6f9cb5f967-8cbzf\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:44.893441 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.893413 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-model-cache\") pod \"test-model-b2fc9f-kserve-6f9cb5f967-8cbzf\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:44.895739 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.895713 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-tls-certs\") pod \"test-model-b2fc9f-kserve-6f9cb5f967-8cbzf\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:44.895930 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.895911 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-dshm\") pod \"test-model-b2fc9f-kserve-6f9cb5f967-8cbzf\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:44.917369 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:44.917340 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4w8g\" (UniqueName: \"kubernetes.io/projected/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-kube-api-access-b4w8g\") pod \"test-model-b2fc9f-kserve-6f9cb5f967-8cbzf\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:45.046953 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:45.045119 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:45.475249 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:58:45.475218 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd26e69c6_ae7f_4fcd_895f_4dcd81062d67.slice/crio-0b6fa64ccbfaeb38d95126731fe5e3ebdd480640901163c01574827daa5e3a1f WatchSource:0}: Error finding container 0b6fa64ccbfaeb38d95126731fe5e3ebdd480640901163c01574827daa5e3a1f: Status 404 returned error can't find the container with id 0b6fa64ccbfaeb38d95126731fe5e3ebdd480640901163c01574827daa5e3a1f Nov 02 17:58:45.476430 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:45.476406 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf"] Nov 02 17:58:46.316317 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:46.316273 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" event={"ID":"d26e69c6-ae7f-4fcd-895f-4dcd81062d67","Type":"ContainerStarted","Data":"7aed4f05b563fc21f2372957a370390691fa5d1caf9912ce62057c0db3f5b3de"} Nov 02 17:58:46.316804 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:46.316329 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" event={"ID":"d26e69c6-ae7f-4fcd-895f-4dcd81062d67","Type":"ContainerStarted","Data":"0b6fa64ccbfaeb38d95126731fe5e3ebdd480640901163c01574827daa5e3a1f"} Nov 02 17:58:46.363468 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:46.363413 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" podStartSLOduration=2.363396121 podStartE2EDuration="2.363396121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:46.358507595 +0000 UTC m=+1744.979867371" watchObservedRunningTime="2026-11-02 17:58:46.363396121 +0000 UTC m=+1744.984755897" Nov 02 17:58:47.046268 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:47.046234 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:49.296358 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:49.296328 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" Nov 02 17:58:51.773010 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:58:51.772974 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:58:55.045910 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:55.045825 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:55.058580 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:55.058550 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:58:55.412035 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:58:55.412005 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 17:59:09.628547 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.628505 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6"] Nov 02 17:59:09.633107 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.633084 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.671651 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.671618 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6"] Nov 02 17:59:09.727495 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.727450 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.727674 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.727586 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.727674 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.727626 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.727766 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.727712 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.727871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.727858 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.727922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.727888 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrcct\" (UniqueName: \"kubernetes.io/projected/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-kube-api-access-jrcct\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.728014 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.727966 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.728099 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.728084 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.728144 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.728130 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.731149 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.731124 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2c9b1d-openshift-default-dockercfg-v7bnw\"" Nov 02 17:59:09.829737 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.829705 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.829939 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.829756 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.829939 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.829865 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.829939 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.829904 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrcct\" (UniqueName: \"kubernetes.io/projected/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-kube-api-access-jrcct\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.830112 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.829957 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.830112 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.830046 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.830112 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.830108 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.830279 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.830158 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.830279 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.830162 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.830279 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.830258 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.830279 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.830275 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.830458 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.830349 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.830458 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.830433 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.830812 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.830786 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.832503 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.832484 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.832774 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.832751 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.853383 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.853359 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.853527 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.853506 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrcct\" (UniqueName: \"kubernetes.io/projected/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-kube-api-access-jrcct\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:09.952554 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:09.952462 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:10.296275 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:59:10.296247 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6975e1bd_ced6_4be9_8808_2e6fa6ff2a89.slice/crio-f4abcfd600783a8e0ea1f0a966d4510e0d024c036eace473d752662bc7f0c62f WatchSource:0}: Error finding container f4abcfd600783a8e0ea1f0a966d4510e0d024c036eace473d752662bc7f0c62f: Status 404 returned error can't find the container with id f4abcfd600783a8e0ea1f0a966d4510e0d024c036eace473d752662bc7f0c62f Nov 02 17:59:10.297237 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:10.297209 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6"] Nov 02 17:59:10.299334 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:10.299296 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:59:10.299437 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:10.299366 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:59:10.299437 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:10.299409 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:59:10.562590 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:10.561748 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" event={"ID":"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89","Type":"ContainerStarted","Data":"a0714c31903728472a126cdc344aa293451b24a277b5d4514db7084e125b6af9"} Nov 02 17:59:10.562590 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:10.561828 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" event={"ID":"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89","Type":"ContainerStarted","Data":"f4abcfd600783a8e0ea1f0a966d4510e0d024c036eace473d752662bc7f0c62f"} Nov 02 17:59:10.608877 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:10.608814 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" podStartSLOduration=1.608800409 podStartE2EDuration="1.608800409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:10.607817914 +0000 UTC m=+1769.229177694" watchObservedRunningTime="2026-11-02 17:59:10.608800409 +0000 UTC m=+1769.230160184" Nov 02 17:59:10.952794 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:10.952698 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:11.980058 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:11.980009 4860 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 02 17:59:11.980580 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:11.980099 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" podUID="6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 02 17:59:12.953577 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:12.953543 4860 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 02 17:59:12.953759 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:12.953601 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" podUID="6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 02 17:59:13.953448 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:13.953408 4860 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 02 17:59:13.953988 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:13.953484 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" podUID="6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 02 17:59:14.953422 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:14.953390 4860 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 02 17:59:14.953601 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:14.953458 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" podUID="6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 02 17:59:15.952942 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:15.952903 4860 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 02 17:59:15.953156 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:15.952959 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" podUID="6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 02 17:59:16.958407 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:16.958369 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:16.958922 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:16.958898 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:16.959774 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:16.959754 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 17:59:17.915618 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:17.915580 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm"] Nov 02 17:59:17.922018 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:17.921988 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:17.952388 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:17.952358 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm"] Nov 02 17:59:18.013554 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.013521 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.013554 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.013557 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.014114 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.013638 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7fxt\" (UniqueName: \"kubernetes.io/projected/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-kube-api-access-h7fxt\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.014114 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.013710 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.014114 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.013784 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.014114 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.013894 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.014114 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.013927 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.014386 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.014116 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.014386 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.014167 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.026079 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.026056 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d5b905-openshift-default-dockercfg-vtgjv\"" Nov 02 17:59:18.116156 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.116118 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.116337 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.116297 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.116337 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.116321 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.116417 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.116356 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7fxt\" (UniqueName: \"kubernetes.io/projected/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-kube-api-access-h7fxt\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.116417 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.116380 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.116516 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.116439 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.116569 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.116522 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.116783 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.116753 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.117011 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.116987 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.117117 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.116804 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.117117 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.117047 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.117234 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.117216 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.117273 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.117232 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.117528 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.117507 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.118662 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.118642 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.119437 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.119413 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.140690 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.140665 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.145691 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.145663 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7fxt\" (UniqueName: \"kubernetes.io/projected/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-kube-api-access-h7fxt\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.245679 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.245600 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:18.577629 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:59:18.577601 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedbe40ed_0184_44e4_990c_2b7a3fd51f5f.slice/crio-05bc174f5be43f8bc5a86bd8acce1e47b453d26509954953b46597059ee69525 WatchSource:0}: Error finding container 05bc174f5be43f8bc5a86bd8acce1e47b453d26509954953b46597059ee69525: Status 404 returned error can't find the container with id 05bc174f5be43f8bc5a86bd8acce1e47b453d26509954953b46597059ee69525 Nov 02 17:59:18.578458 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.578434 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm"] Nov 02 17:59:18.580774 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.580740 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:59:18.580911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.580806 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:59:18.580911 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.580836 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:59:18.654336 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:18.654300 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" event={"ID":"edbe40ed-0184-44e4-990c-2b7a3fd51f5f","Type":"ContainerStarted","Data":"05bc174f5be43f8bc5a86bd8acce1e47b453d26509954953b46597059ee69525"} Nov 02 17:59:19.690565 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:19.690527 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" event={"ID":"edbe40ed-0184-44e4-990c-2b7a3fd51f5f","Type":"ContainerStarted","Data":"0ab71876983a48a4176a21bcd725fc54a43cf3027848dd51e64d85da27fde6ac"} Nov 02 17:59:19.775872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:19.773132 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" podStartSLOduration=2.7731096429999997 podStartE2EDuration="2.773109643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:19.751709603 +0000 UTC m=+1778.373069380" watchObservedRunningTime="2026-11-02 17:59:19.773109643 +0000 UTC m=+1778.394469419" Nov 02 17:59:20.246255 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:20.246219 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:20.247812 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:20.247781 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 02 17:59:20.247988 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:20.247844 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" podUID="edbe40ed-0184-44e4-990c-2b7a3fd51f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 02 17:59:21.286402 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:21.286364 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 02 17:59:21.286991 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:21.286959 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" podUID="edbe40ed-0184-44e4-990c-2b7a3fd51f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 02 17:59:21.800709 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:59:21.800670 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 17:59:22.201502 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.201442 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl"] Nov 02 17:59:22.206670 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.206616 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.235073 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.234352 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl"] Nov 02 17:59:22.246323 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.245409 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.246323 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.245543 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.246323 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.245594 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.246323 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.245673 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdxq2\" (UniqueName: \"kubernetes.io/projected/151de229-e8e3-4bc3-8443-b12cfca9aab8-kube-api-access-jdxq2\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.246323 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.245770 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.246323 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.245815 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.246323 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.245868 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.246323 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.245975 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151de229-e8e3-4bc3-8443-b12cfca9aab8-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.246323 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.246106 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.247123 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.247099 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 02 17:59:22.247217 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.247161 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" podUID="edbe40ed-0184-44e4-990c-2b7a3fd51f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 02 17:59:22.274875 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.272154 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-96768b-openshift-default-dockercfg-v8qql\"" Nov 02 17:59:22.347279 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.347248 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.347782 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.347344 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.347782 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.347391 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.347782 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.347449 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151de229-e8e3-4bc3-8443-b12cfca9aab8-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.347782 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.347541 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.347782 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.347721 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.347782 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.347741 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.348157 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.347871 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.348157 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.347927 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.348157 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.347956 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdxq2\" (UniqueName: \"kubernetes.io/projected/151de229-e8e3-4bc3-8443-b12cfca9aab8-kube-api-access-jdxq2\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.348157 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.347997 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.348393 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.348346 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151de229-e8e3-4bc3-8443-b12cfca9aab8-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.348547 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.348520 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.349033 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.349009 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.350780 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.350756 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.351176 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.351152 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.386464 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.386430 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.386701 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.386575 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdxq2\" (UniqueName: \"kubernetes.io/projected/151de229-e8e3-4bc3-8443-b12cfca9aab8-kube-api-access-jdxq2\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.530835 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.530744 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:22.900302 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.900264 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl"] Nov 02 17:59:22.902009 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:59:22.901927 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151de229_e8e3_4bc3_8443_b12cfca9aab8.slice/crio-76ca239226d5459b9df1da4c4a914e4cd89a7bac3f6705ced7be79861f0616b1 WatchSource:0}: Error finding container 76ca239226d5459b9df1da4c4a914e4cd89a7bac3f6705ced7be79861f0616b1: Status 404 returned error can't find the container with id 76ca239226d5459b9df1da4c4a914e4cd89a7bac3f6705ced7be79861f0616b1 Nov 02 17:59:22.905503 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.905473 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:59:22.905687 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.905667 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:59:22.905793 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:22.905777 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:59:23.134950 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.134913 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w"] Nov 02 17:59:23.145233 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.145199 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.163993 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.163904 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w"] Nov 02 17:59:23.164493 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.164470 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d5b905\"/\"kube-root-ca.crt\"" Nov 02 17:59:23.211103 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.210258 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d5b905\"/\"openshift-service-ca.crt\"" Nov 02 17:59:23.211786 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.211381 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d5b905\"/\"test-model-d5b905-kserve-self-signed-certs\"" Nov 02 17:59:23.223748 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.223206 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d5b905\"/\"default-dockercfg-6lmrh\"" Nov 02 17:59:23.308780 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.308742 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-tmp-dir\") pod \"test-model-d5b905-kserve-545db554fc-8mc9w\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.316054 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.316023 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-model-cache\") pod \"test-model-d5b905-kserve-545db554fc-8mc9w\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.318011 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.316461 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9vmg\" (UniqueName: \"kubernetes.io/projected/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-kube-api-access-k9vmg\") pod \"test-model-d5b905-kserve-545db554fc-8mc9w\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.318011 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.316557 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-dshm\") pod \"test-model-d5b905-kserve-545db554fc-8mc9w\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.318011 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.316654 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-tls-certs\") pod \"test-model-d5b905-kserve-545db554fc-8mc9w\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.318011 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.316689 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-home\") pod \"test-model-d5b905-kserve-545db554fc-8mc9w\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.322883 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.322839 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 02 17:59:23.323831 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.323782 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" podUID="edbe40ed-0184-44e4-990c-2b7a3fd51f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 02 17:59:23.435919 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.434701 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-tmp-dir\") pod \"test-model-d5b905-kserve-545db554fc-8mc9w\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.435919 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.434945 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-model-cache\") pod \"test-model-d5b905-kserve-545db554fc-8mc9w\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.435919 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.435344 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9vmg\" (UniqueName: \"kubernetes.io/projected/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-kube-api-access-k9vmg\") pod \"test-model-d5b905-kserve-545db554fc-8mc9w\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.435919 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.435425 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-dshm\") pod \"test-model-d5b905-kserve-545db554fc-8mc9w\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.435919 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.435518 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-tls-certs\") pod \"test-model-d5b905-kserve-545db554fc-8mc9w\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.435919 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.435549 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-home\") pod \"test-model-d5b905-kserve-545db554fc-8mc9w\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.436665 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.436126 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-home\") pod \"test-model-d5b905-kserve-545db554fc-8mc9w\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.436665 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.436389 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-tmp-dir\") pod \"test-model-d5b905-kserve-545db554fc-8mc9w\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.436772 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.436675 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-model-cache\") pod \"test-model-d5b905-kserve-545db554fc-8mc9w\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.470530 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.470493 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9vmg\" (UniqueName: \"kubernetes.io/projected/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-kube-api-access-k9vmg\") pod \"test-model-d5b905-kserve-545db554fc-8mc9w\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.471034 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.470968 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-dshm\") pod \"test-model-d5b905-kserve-545db554fc-8mc9w\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.474658 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.474631 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-tls-certs\") pod \"test-model-d5b905-kserve-545db554fc-8mc9w\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.624376 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.623256 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:23.758685 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.754873 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" event={"ID":"151de229-e8e3-4bc3-8443-b12cfca9aab8","Type":"ContainerStarted","Data":"4c74414f378a26764acdd5f5aa77f5a7840a3d7eee8185b7077c602b64fdb46e"} Nov 02 17:59:23.758685 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.754941 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" event={"ID":"151de229-e8e3-4bc3-8443-b12cfca9aab8","Type":"ContainerStarted","Data":"76ca239226d5459b9df1da4c4a914e4cd89a7bac3f6705ced7be79861f0616b1"} Nov 02 17:59:23.827423 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:23.825246 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" podStartSLOduration=1.825228482 podStartE2EDuration="1.825228482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:23.808340616 +0000 UTC m=+1782.429700392" watchObservedRunningTime="2026-11-02 17:59:23.825228482 +0000 UTC m=+1782.446588258" Nov 02 17:59:24.173904 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.152766 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w"] Nov 02 17:59:24.251963 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.251382 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 02 17:59:24.251963 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.251441 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" podUID="edbe40ed-0184-44e4-990c-2b7a3fd51f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 02 17:59:24.531407 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.531276 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:24.533314 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.533065 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 02 17:59:24.533314 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.533104 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" podUID="151de229-e8e3-4bc3-8443-b12cfca9aab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 02 17:59:24.712927 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.712105 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr"] Nov 02 17:59:24.742361 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.741831 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.747833 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.745893 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr"] Nov 02 17:59:24.835749 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.835705 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.836029 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.835997 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.840531 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.840496 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9293d64-b25e-4076-b707-fc65717da6a8-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.840784 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.840764 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9293d64-b25e-4076-b707-fc65717da6a8-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.845023 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.844996 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" event={"ID":"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff","Type":"ContainerStarted","Data":"fbc0a8e64fd5342ec7b4f7f53020304bd3c689dbd2d646d4216007152a1e65f8"} Nov 02 17:59:24.845190 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.845173 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" event={"ID":"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff","Type":"ContainerStarted","Data":"a23bd586349777c5091671301d82872bba50d7e0c07625009d144d0f5179f21c"} Nov 02 17:59:24.846124 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.845838 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdwvn\" (UniqueName: \"kubernetes.io/projected/e9293d64-b25e-4076-b707-fc65717da6a8-kube-api-access-rdwvn\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.846595 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.846576 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.847194 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.847172 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9293d64-b25e-4076-b707-fc65717da6a8-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.847341 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.847324 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.847462 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.847432 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.944891 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.944237 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" podStartSLOduration=1.9442160240000002 podStartE2EDuration="1.944216024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:24.908032789 +0000 UTC m=+1783.529392566" watchObservedRunningTime="2026-11-02 17:59:24.944216024 +0000 UTC m=+1783.565575801" Nov 02 17:59:24.948447 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.948420 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.949451 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.949423 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.949585 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.949466 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9293d64-b25e-4076-b707-fc65717da6a8-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.949585 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.949521 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9293d64-b25e-4076-b707-fc65717da6a8-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.949694 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.949593 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdwvn\" (UniqueName: \"kubernetes.io/projected/e9293d64-b25e-4076-b707-fc65717da6a8-kube-api-access-rdwvn\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.949694 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.949678 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.949805 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.949752 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9293d64-b25e-4076-b707-fc65717da6a8-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.949805 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.949789 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.949956 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.949825 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.951133 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.951106 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9293d64-b25e-4076-b707-fc65717da6a8-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.951590 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.951534 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.951986 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.951947 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.952098 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.952079 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.952278 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.952257 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.953166 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.953150 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9293d64-b25e-4076-b707-fc65717da6a8-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:24.996936 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:24.969606 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:25.007553 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:25.007507 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9293d64-b25e-4076-b707-fc65717da6a8-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:25.046955 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:25.046922 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdwvn\" (UniqueName: \"kubernetes.io/projected/e9293d64-b25e-4076-b707-fc65717da6a8-kube-api-access-rdwvn\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:25.149082 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:25.146025 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:25.245959 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:25.245919 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 02 17:59:25.246203 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:25.245979 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" podUID="edbe40ed-0184-44e4-990c-2b7a3fd51f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 02 17:59:25.532100 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:25.532013 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 02 17:59:25.532100 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:25.532071 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" podUID="151de229-e8e3-4bc3-8443-b12cfca9aab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 02 17:59:25.594916 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:25.594878 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:59:25.595075 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:25.594982 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:59:25.595075 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:25.595033 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:59:25.599937 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:25.598717 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr"] Nov 02 17:59:25.624423 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:25.623957 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:25.876530 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:25.876495 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" event={"ID":"e9293d64-b25e-4076-b707-fc65717da6a8","Type":"ContainerStarted","Data":"4ac94621743bb320f5be5d0c9c03c21a8977f1c6ea50e0e744011f653dc35b5b"} Nov 02 17:59:25.877618 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:25.877597 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" event={"ID":"e9293d64-b25e-4076-b707-fc65717da6a8","Type":"ContainerStarted","Data":"36e05a6ae1db066818a27a37c26afed9a4d9d20fd79ff7d5d6218cdc68c1c517"} Nov 02 17:59:25.933579 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:25.933517 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" podStartSLOduration=1.93349992 podStartE2EDuration="1.93349992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:25.927179539 +0000 UTC m=+1784.548539316" watchObservedRunningTime="2026-11-02 17:59:25.93349992 +0000 UTC m=+1784.554859696" Nov 02 17:59:26.146636 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:26.146539 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:26.148822 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:26.148787 4860 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 02 17:59:26.149005 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:26.148862 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" podUID="e9293d64-b25e-4076-b707-fc65717da6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 02 17:59:26.252957 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:26.252918 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:26.253452 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:26.253420 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:26.261871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:26.258078 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 17:59:26.533202 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:26.532770 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 02 17:59:26.533202 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:26.532833 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" podUID="151de229-e8e3-4bc3-8443-b12cfca9aab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 02 17:59:27.147205 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:27.147171 4860 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 02 17:59:27.147422 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:27.147227 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" podUID="e9293d64-b25e-4076-b707-fc65717da6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 02 17:59:27.531975 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:27.531880 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 02 17:59:27.531975 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:27.531949 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" podUID="151de229-e8e3-4bc3-8443-b12cfca9aab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 02 17:59:28.146982 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:28.146614 4860 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 02 17:59:28.146982 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:28.146665 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" podUID="e9293d64-b25e-4076-b707-fc65717da6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 02 17:59:28.538287 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:28.538191 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 02 17:59:28.538287 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:28.538266 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" podUID="151de229-e8e3-4bc3-8443-b12cfca9aab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 02 17:59:29.157871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:29.150181 4860 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 02 17:59:29.157871 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:29.150244 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" podUID="e9293d64-b25e-4076-b707-fc65717da6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 02 17:59:29.538131 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:29.538052 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 02 17:59:29.539678 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:29.538922 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" podUID="151de229-e8e3-4bc3-8443-b12cfca9aab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 02 17:59:30.147352 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:30.147320 4860 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 02 17:59:30.147521 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:30.147370 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" podUID="e9293d64-b25e-4076-b707-fc65717da6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 02 17:59:30.536914 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:30.536827 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:30.537341 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:30.537321 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:30.538231 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:30.538209 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 17:59:31.147236 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:31.147199 4860 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 02 17:59:31.147408 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:31.147261 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" podUID="e9293d64-b25e-4076-b707-fc65717da6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 02 17:59:32.147605 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:32.147571 4860 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 02 17:59:32.148055 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:32.147636 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" podUID="e9293d64-b25e-4076-b707-fc65717da6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 02 17:59:32.993561 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:32.993522 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5"] Nov 02 17:59:32.999389 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:32.999363 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.042046 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.042004 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5"] Nov 02 17:59:33.067656 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.067620 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.067899 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.067678 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.067998 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.067890 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.067998 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.067932 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.068129 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.068109 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.068194 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.068167 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ef1178-de19-4f53-9646-f4ac3ed52f57-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.068251 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.068200 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.068320 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.068276 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.068376 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.068359 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g79gx\" (UniqueName: \"kubernetes.io/projected/95ef1178-de19-4f53-9646-f4ac3ed52f57-kube-api-access-g79gx\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.152139 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.152109 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:33.152578 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.152462 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:33.153058 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.153031 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 17:59:33.169951 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.169924 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.170097 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.170006 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g79gx\" (UniqueName: \"kubernetes.io/projected/95ef1178-de19-4f53-9646-f4ac3ed52f57-kube-api-access-g79gx\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.170176 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.170159 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.170242 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.170222 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.170345 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.170329 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.170445 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.170360 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.170445 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.170379 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.170558 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.170471 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.170558 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.170537 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.170669 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.170610 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ef1178-de19-4f53-9646-f4ac3ed52f57-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.170669 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.170630 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.170669 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.170645 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.170967 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.170948 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.171323 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.171301 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ef1178-de19-4f53-9646-f4ac3ed52f57-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.172461 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.172440 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.172840 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.172823 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.197589 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.197570 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.199594 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.199572 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g79gx\" (UniqueName: \"kubernetes.io/projected/95ef1178-de19-4f53-9646-f4ac3ed52f57-kube-api-access-g79gx\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.317766 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.317727 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:33.665912 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.662186 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:33.673696 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.673664 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:33.690958 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.690922 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:59:33.691097 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.691011 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:59:33.691097 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.691064 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:59:33.753423 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:33.753376 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5"] Nov 02 17:59:34.138104 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:34.138067 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" event={"ID":"95ef1178-de19-4f53-9646-f4ac3ed52f57","Type":"ContainerStarted","Data":"416d2d45c71688cdab5ab65c2b44cf698c60c35f49ea83ad91929a47faf6a4e5"} Nov 02 17:59:34.138309 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:34.138125 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" event={"ID":"95ef1178-de19-4f53-9646-f4ac3ed52f57","Type":"ContainerStarted","Data":"2a29ed4c926ee4f10e6cd7cb0b7ead348a52dc5213282580d792463319075fa8"} Nov 02 17:59:34.150552 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:34.150522 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 17:59:34.194215 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:34.191684 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" podStartSLOduration=2.191664619 podStartE2EDuration="2.191664619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:34.184114872 +0000 UTC m=+1792.805474648" watchObservedRunningTime="2026-11-02 17:59:34.191664619 +0000 UTC m=+1792.813024397" Nov 02 17:59:34.318020 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:34.317984 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:34.319583 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:34.319557 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 17:59:34.319718 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:34.319605 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" podUID="95ef1178-de19-4f53-9646-f4ac3ed52f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 17:59:35.318099 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:35.318065 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 17:59:35.318496 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:35.318128 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" podUID="95ef1178-de19-4f53-9646-f4ac3ed52f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 17:59:36.100607 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.100566 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx"] Nov 02 17:59:36.107078 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.107044 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.126765 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.125338 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx"] Nov 02 17:59:36.129460 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.129432 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2c9b1d\"/\"default-dockercfg-lwl2r\"" Nov 02 17:59:36.129556 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.129496 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2c9b1d\"/\"test-model-2c9b1d-kserve-self-signed-certs\"" Nov 02 17:59:36.137674 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.137644 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2c9b1d\"/\"kube-root-ca.crt\"" Nov 02 17:59:36.137905 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.137883 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2c9b1d\"/\"openshift-service-ca.crt\"" Nov 02 17:59:36.206993 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.206958 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-dshm\") pod \"test-model-2c9b1d-kserve-75f7cb9f89-9g7gx\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.207193 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.207090 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpjg2\" (UniqueName: \"kubernetes.io/projected/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-kube-api-access-fpjg2\") pod \"test-model-2c9b1d-kserve-75f7cb9f89-9g7gx\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.207353 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.207327 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-tls-certs\") pod \"test-model-2c9b1d-kserve-75f7cb9f89-9g7gx\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.207408 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.207394 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-home\") pod \"test-model-2c9b1d-kserve-75f7cb9f89-9g7gx\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.207471 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.207454 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-tmp-dir\") pod \"test-model-2c9b1d-kserve-75f7cb9f89-9g7gx\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.207549 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.207535 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-model-cache\") pod \"test-model-2c9b1d-kserve-75f7cb9f89-9g7gx\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.309221 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.309174 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-tls-certs\") pod \"test-model-2c9b1d-kserve-75f7cb9f89-9g7gx\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.309438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.309320 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-home\") pod \"test-model-2c9b1d-kserve-75f7cb9f89-9g7gx\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.309438 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.309382 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-tmp-dir\") pod \"test-model-2c9b1d-kserve-75f7cb9f89-9g7gx\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.309557 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.309469 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-model-cache\") pod \"test-model-2c9b1d-kserve-75f7cb9f89-9g7gx\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.309557 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.309541 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-dshm\") pod \"test-model-2c9b1d-kserve-75f7cb9f89-9g7gx\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.309667 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.309650 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpjg2\" (UniqueName: \"kubernetes.io/projected/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-kube-api-access-fpjg2\") pod \"test-model-2c9b1d-kserve-75f7cb9f89-9g7gx\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.310153 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.309775 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-home\") pod \"test-model-2c9b1d-kserve-75f7cb9f89-9g7gx\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.310153 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.309806 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-tmp-dir\") pod \"test-model-2c9b1d-kserve-75f7cb9f89-9g7gx\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.310153 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.310102 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-model-cache\") pod \"test-model-2c9b1d-kserve-75f7cb9f89-9g7gx\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.314390 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.314187 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-dshm\") pod \"test-model-2c9b1d-kserve-75f7cb9f89-9g7gx\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.314508 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.314408 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-tls-certs\") pod \"test-model-2c9b1d-kserve-75f7cb9f89-9g7gx\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.318531 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.318506 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 17:59:36.318841 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.318559 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" podUID="95ef1178-de19-4f53-9646-f4ac3ed52f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 17:59:36.334806 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.334774 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpjg2\" (UniqueName: \"kubernetes.io/projected/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-kube-api-access-fpjg2\") pod \"test-model-2c9b1d-kserve-75f7cb9f89-9g7gx\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.435936 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.435150 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:36.989456 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:59:36.989419 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d7782ec_d1ac_4fc7_9ce6_2b8866af3d10.slice/crio-9c5b5697ffe50cf7a9a0421e38e22cb8ad7a829fb63a97c11fdcceab85a379b2 WatchSource:0}: Error finding container 9c5b5697ffe50cf7a9a0421e38e22cb8ad7a829fb63a97c11fdcceab85a379b2: Status 404 returned error can't find the container with id 9c5b5697ffe50cf7a9a0421e38e22cb8ad7a829fb63a97c11fdcceab85a379b2 Nov 02 17:59:36.991600 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:36.991573 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx"] Nov 02 17:59:37.215511 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.214258 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" event={"ID":"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10","Type":"ContainerStarted","Data":"492dec3242c3125df999443a8ec98a1bd9b84f229d8d436dc13c682e37d4c1a6"} Nov 02 17:59:37.215511 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.214334 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" event={"ID":"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10","Type":"ContainerStarted","Data":"9c5b5697ffe50cf7a9a0421e38e22cb8ad7a829fb63a97c11fdcceab85a379b2"} Nov 02 17:59:37.252471 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.252392 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh"] Nov 02 17:59:37.265872 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.264214 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" podStartSLOduration=1.264194407 podStartE2EDuration="1.264194407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:37.262736652 +0000 UTC m=+1795.884096434" watchObservedRunningTime="2026-11-02 17:59:37.264194407 +0000 UTC m=+1795.885554184" Nov 02 17:59:37.299442 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.299197 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.299442 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.299331 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh"] Nov 02 17:59:37.319462 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.319432 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 17:59:37.320050 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.320016 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" podUID="95ef1178-de19-4f53-9646-f4ac3ed52f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 17:59:37.349394 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.349363 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.349550 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.349448 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.349550 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.349473 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.349550 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.349520 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.349550 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.349538 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.349702 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.349602 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.349702 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.349637 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.349702 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.349663 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsdtt\" (UniqueName: \"kubernetes.io/projected/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-kube-api-access-tsdtt\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.349804 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.349713 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.439712 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.439674 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:37.451610 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.451108 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.451610 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.451153 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.451610 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.451202 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.451610 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.451218 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.451610 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.451281 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.451610 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.451315 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.451610 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.451338 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsdtt\" (UniqueName: \"kubernetes.io/projected/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-kube-api-access-tsdtt\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.451610 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.451389 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.451610 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.451491 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.453036 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.452891 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.453036 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.452905 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.453036 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.452991 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.453036 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.453011 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.453353 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.453321 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.476887 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.476128 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.477888 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.477789 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.489095 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.489065 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.489340 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.489219 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsdtt\" (UniqueName: \"kubernetes.io/projected/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-kube-api-access-tsdtt\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:37.638249 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:37.638214 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:38.187885 ip-10-0-136-153 kubenswrapper[4860]: W1102 17:59:38.182686 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2fd7d0_d1d9_4087_9d88_25142f4a6b65.slice/crio-5fe51f59c725ddd1dcfdcebea620964e7a4aa0bc092a34615314449fd34f7350 WatchSource:0}: Error finding container 5fe51f59c725ddd1dcfdcebea620964e7a4aa0bc092a34615314449fd34f7350: Status 404 returned error can't find the container with id 5fe51f59c725ddd1dcfdcebea620964e7a4aa0bc092a34615314449fd34f7350 Nov 02 17:59:38.204025 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:38.203197 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:59:38.204025 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:38.203352 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:59:38.204025 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:38.203404 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:59:38.217645 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:38.207191 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh"] Nov 02 17:59:38.325495 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:38.325223 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 17:59:38.327284 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:38.327246 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" podUID="95ef1178-de19-4f53-9646-f4ac3ed52f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 17:59:38.501201 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:38.501106 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" event={"ID":"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65","Type":"ContainerStarted","Data":"5fe51f59c725ddd1dcfdcebea620964e7a4aa0bc092a34615314449fd34f7350"} Nov 02 17:59:39.319214 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:39.319175 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 17:59:39.319411 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:39.319237 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" podUID="95ef1178-de19-4f53-9646-f4ac3ed52f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 17:59:39.512755 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:39.512716 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" event={"ID":"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65","Type":"ContainerStarted","Data":"31b4398bf86c26c964b4c307cb44eb619037686ca5b55915388dee728665d4d9"} Nov 02 17:59:39.616602 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:39.616482 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" podStartSLOduration=2.6164611840000003 podStartE2EDuration="2.616461184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:39.612202037 +0000 UTC m=+1798.233561851" watchObservedRunningTime="2026-11-02 17:59:39.616461184 +0000 UTC m=+1798.237820977" Nov 02 17:59:39.638744 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:39.638712 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:39.641272 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:39.640975 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 02 17:59:39.641272 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:39.641030 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" podUID="8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 02 17:59:40.318624 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:40.318586 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 17:59:40.318834 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:40.318652 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" podUID="95ef1178-de19-4f53-9646-f4ac3ed52f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 17:59:40.639507 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:40.639417 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 02 17:59:40.639507 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:40.639475 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" podUID="8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 02 17:59:41.483823 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:41.483785 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:41.485028 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:41.485004 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:41.486537 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:41.486405 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 17:59:41.638803 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:41.638770 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 02 17:59:41.638991 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:41.638836 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" podUID="8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 02 17:59:41.901036 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:41.901003 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 17:59:41.913063 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:41.913034 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 17:59:41.918654 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:41.918623 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 17:59:41.933356 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:41.933323 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 17:59:42.639584 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:42.639542 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 02 17:59:42.639788 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:42.639616 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" podUID="8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 02 17:59:43.639574 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:43.639537 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 02 17:59:43.640102 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:43.639594 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" podUID="8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 02 17:59:44.638617 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:44.638586 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 02 17:59:44.638787 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:44.638638 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" podUID="8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 02 17:59:45.639247 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:45.639210 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 02 17:59:45.639638 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:45.639266 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" podUID="8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 02 17:59:46.435782 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:46.435745 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:46.457314 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:46.457279 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:46.619596 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:46.619563 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 17:59:46.643927 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:46.643898 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:46.644385 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:46.644176 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:46.644736 ip-10-0-136-153 kubenswrapper[4860]: I1102 17:59:46.644711 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 17:59:51.771780 ip-10-0-136-153 kubenswrapper[4860]: E1102 17:59:51.771742 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:00:00.389872 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.386280 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h"] Nov 02 18:00:00.460917 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.456910 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" Nov 02 18:00:00.486930 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.477426 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv"] Nov 02 18:00:00.486930 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.480600 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 18:00:00.528074 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.528046 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m"] Nov 02 18:00:00.530958 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.530934 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" Nov 02 18:00:00.537756 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.537731 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr"] Nov 02 18:00:00.556423 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.538652 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" Nov 02 18:00:00.556423 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.544487 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h"] Nov 02 18:00:00.556423 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.544542 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv"] Nov 02 18:00:00.556423 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.550115 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv"] Nov 02 18:00:00.556423 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.550153 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr"] Nov 02 18:00:00.556423 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.550181 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6"] Nov 02 18:00:00.556423 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.550629 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" Nov 02 18:00:00.556423 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.556205 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/487dac52-8fbc-425e-af6c-3db7403709ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h\" (UID: \"487dac52-8fbc-425e-af6c-3db7403709ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" Nov 02 18:00:00.556423 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.556207 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" Nov 02 18:00:00.557379 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.557293 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb6ww\" (UniqueName: \"kubernetes.io/projected/487dac52-8fbc-425e-af6c-3db7403709ff-kube-api-access-vb6ww\") pod \"maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h\" (UID: \"487dac52-8fbc-425e-af6c-3db7403709ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" Nov 02 18:00:00.558669 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.558650 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv"] Nov 02 18:00:00.558878 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.558827 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6"] Nov 02 18:00:00.559097 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.558762 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" Nov 02 18:00:00.568031 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.568008 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m"] Nov 02 18:00:00.660290 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.660197 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb6ww\" (UniqueName: \"kubernetes.io/projected/487dac52-8fbc-425e-af6c-3db7403709ff-kube-api-access-vb6ww\") pod \"maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h\" (UID: \"487dac52-8fbc-425e-af6c-3db7403709ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" Nov 02 18:00:00.660290 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.660261 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79vh5\" (UniqueName: \"kubernetes.io/projected/65b76941-3104-4e66-a6b0-52a20c9eb4f5-kube-api-access-79vh5\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv\" (UID: \"65b76941-3104-4e66-a6b0-52a20c9eb4f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" Nov 02 18:00:00.660586 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.660324 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ffcc622-4641-4348-ad7a-88797d8b2469-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6\" (UID: \"0ffcc622-4641-4348-ad7a-88797d8b2469\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" Nov 02 18:00:00.660586 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.660345 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmbnh\" (UniqueName: \"kubernetes.io/projected/0ffcc622-4641-4348-ad7a-88797d8b2469-kube-api-access-tmbnh\") pod \"maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6\" (UID: \"0ffcc622-4641-4348-ad7a-88797d8b2469\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" Nov 02 18:00:00.660586 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.660489 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2366ff-2ee5-4ef5-b9f8-5605a1337924-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m\" (UID: \"0d2366ff-2ee5-4ef5-b9f8-5605a1337924\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" Nov 02 18:00:00.660586 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.660531 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djs6b\" (UniqueName: \"kubernetes.io/projected/0d2366ff-2ee5-4ef5-b9f8-5605a1337924-kube-api-access-djs6b\") pod \"maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m\" (UID: \"0d2366ff-2ee5-4ef5-b9f8-5605a1337924\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" Nov 02 18:00:00.660809 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.660586 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1cc711d-aed9-4bdc-84b8-795dfc52a7bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv\" (UID: \"f1cc711d-aed9-4bdc-84b8-795dfc52a7bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" Nov 02 18:00:00.660809 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.660630 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/487dac52-8fbc-425e-af6c-3db7403709ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h\" (UID: \"487dac52-8fbc-425e-af6c-3db7403709ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" Nov 02 18:00:00.660809 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.660697 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7zbx\" (UniqueName: \"kubernetes.io/projected/f1cc711d-aed9-4bdc-84b8-795dfc52a7bd-kube-api-access-w7zbx\") pod \"maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv\" (UID: \"f1cc711d-aed9-4bdc-84b8-795dfc52a7bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" Nov 02 18:00:00.660809 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.660754 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk6h5\" (UniqueName: \"kubernetes.io/projected/54f4f507-8d94-412d-9189-7ac2a4531f01-kube-api-access-pk6h5\") pod \"maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr\" (UID: \"54f4f507-8d94-412d-9189-7ac2a4531f01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" Nov 02 18:00:00.661038 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.660812 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b76941-3104-4e66-a6b0-52a20c9eb4f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv\" (UID: \"65b76941-3104-4e66-a6b0-52a20c9eb4f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" Nov 02 18:00:00.661829 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.661756 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f4f507-8d94-412d-9189-7ac2a4531f01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr\" (UID: \"54f4f507-8d94-412d-9189-7ac2a4531f01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" Nov 02 18:00:00.670159 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.667244 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/487dac52-8fbc-425e-af6c-3db7403709ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h\" (UID: \"487dac52-8fbc-425e-af6c-3db7403709ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" Nov 02 18:00:00.701052 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.701017 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb6ww\" (UniqueName: \"kubernetes.io/projected/487dac52-8fbc-425e-af6c-3db7403709ff-kube-api-access-vb6ww\") pod \"maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h\" (UID: \"487dac52-8fbc-425e-af6c-3db7403709ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" Nov 02 18:00:00.790874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.785961 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f4f507-8d94-412d-9189-7ac2a4531f01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr\" (UID: \"54f4f507-8d94-412d-9189-7ac2a4531f01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" Nov 02 18:00:00.790874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.786037 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79vh5\" (UniqueName: \"kubernetes.io/projected/65b76941-3104-4e66-a6b0-52a20c9eb4f5-kube-api-access-79vh5\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv\" (UID: \"65b76941-3104-4e66-a6b0-52a20c9eb4f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" Nov 02 18:00:00.790874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.786083 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ffcc622-4641-4348-ad7a-88797d8b2469-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6\" (UID: \"0ffcc622-4641-4348-ad7a-88797d8b2469\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" Nov 02 18:00:00.790874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.786111 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmbnh\" (UniqueName: \"kubernetes.io/projected/0ffcc622-4641-4348-ad7a-88797d8b2469-kube-api-access-tmbnh\") pod \"maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6\" (UID: \"0ffcc622-4641-4348-ad7a-88797d8b2469\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" Nov 02 18:00:00.790874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.786246 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2366ff-2ee5-4ef5-b9f8-5605a1337924-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m\" (UID: \"0d2366ff-2ee5-4ef5-b9f8-5605a1337924\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" Nov 02 18:00:00.790874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.786288 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djs6b\" (UniqueName: \"kubernetes.io/projected/0d2366ff-2ee5-4ef5-b9f8-5605a1337924-kube-api-access-djs6b\") pod \"maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m\" (UID: \"0d2366ff-2ee5-4ef5-b9f8-5605a1337924\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" Nov 02 18:00:00.790874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.786337 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1cc711d-aed9-4bdc-84b8-795dfc52a7bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv\" (UID: \"f1cc711d-aed9-4bdc-84b8-795dfc52a7bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" Nov 02 18:00:00.790874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.786410 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7zbx\" (UniqueName: \"kubernetes.io/projected/f1cc711d-aed9-4bdc-84b8-795dfc52a7bd-kube-api-access-w7zbx\") pod \"maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv\" (UID: \"f1cc711d-aed9-4bdc-84b8-795dfc52a7bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" Nov 02 18:00:00.790874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.786463 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk6h5\" (UniqueName: \"kubernetes.io/projected/54f4f507-8d94-412d-9189-7ac2a4531f01-kube-api-access-pk6h5\") pod \"maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr\" (UID: \"54f4f507-8d94-412d-9189-7ac2a4531f01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" Nov 02 18:00:00.790874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.786518 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b76941-3104-4e66-a6b0-52a20c9eb4f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv\" (UID: \"65b76941-3104-4e66-a6b0-52a20c9eb4f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" Nov 02 18:00:00.790874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.787521 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b76941-3104-4e66-a6b0-52a20c9eb4f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv\" (UID: \"65b76941-3104-4e66-a6b0-52a20c9eb4f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" Nov 02 18:00:00.790874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.788543 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f4f507-8d94-412d-9189-7ac2a4531f01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr\" (UID: \"54f4f507-8d94-412d-9189-7ac2a4531f01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" Nov 02 18:00:00.790874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.790086 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ffcc622-4641-4348-ad7a-88797d8b2469-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6\" (UID: \"0ffcc622-4641-4348-ad7a-88797d8b2469\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" Nov 02 18:00:00.791728 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.791200 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2366ff-2ee5-4ef5-b9f8-5605a1337924-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m\" (UID: \"0d2366ff-2ee5-4ef5-b9f8-5605a1337924\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" Nov 02 18:00:00.792552 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.792063 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1cc711d-aed9-4bdc-84b8-795dfc52a7bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv\" (UID: \"f1cc711d-aed9-4bdc-84b8-795dfc52a7bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" Nov 02 18:00:00.846870 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.842018 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" Nov 02 18:00:00.904046 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.903999 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7zbx\" (UniqueName: \"kubernetes.io/projected/f1cc711d-aed9-4bdc-84b8-795dfc52a7bd-kube-api-access-w7zbx\") pod \"maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv\" (UID: \"f1cc711d-aed9-4bdc-84b8-795dfc52a7bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" Nov 02 18:00:00.941830 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.941750 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk6h5\" (UniqueName: \"kubernetes.io/projected/54f4f507-8d94-412d-9189-7ac2a4531f01-kube-api-access-pk6h5\") pod \"maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr\" (UID: \"54f4f507-8d94-412d-9189-7ac2a4531f01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" Nov 02 18:00:00.942771 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.942719 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79vh5\" (UniqueName: \"kubernetes.io/projected/65b76941-3104-4e66-a6b0-52a20c9eb4f5-kube-api-access-79vh5\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv\" (UID: \"65b76941-3104-4e66-a6b0-52a20c9eb4f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" Nov 02 18:00:00.945019 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.944995 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmbnh\" (UniqueName: \"kubernetes.io/projected/0ffcc622-4641-4348-ad7a-88797d8b2469-kube-api-access-tmbnh\") pod \"maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6\" (UID: \"0ffcc622-4641-4348-ad7a-88797d8b2469\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" Nov 02 18:00:00.961448 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:00.958671 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djs6b\" (UniqueName: \"kubernetes.io/projected/0d2366ff-2ee5-4ef5-b9f8-5605a1337924-kube-api-access-djs6b\") pod \"maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m\" (UID: \"0d2366ff-2ee5-4ef5-b9f8-5605a1337924\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" Nov 02 18:00:01.192820 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:01.192704 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" Nov 02 18:00:01.209988 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:01.203053 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" Nov 02 18:00:01.217886 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:01.214500 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" Nov 02 18:00:01.226425 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:01.226100 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" Nov 02 18:00:01.240377 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:01.240210 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" Nov 02 18:00:01.690282 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:00:01.690236 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487dac52_8fbc_425e_af6c_3db7403709ff.slice/crio-ba3b91c304e57217e5b3aa06366ab70b5b0b7ab64154944768869ca152633235 WatchSource:0}: Error finding container ba3b91c304e57217e5b3aa06366ab70b5b0b7ab64154944768869ca152633235: Status 404 returned error can't find the container with id ba3b91c304e57217e5b3aa06366ab70b5b0b7ab64154944768869ca152633235 Nov 02 18:00:01.690707 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:01.690314 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h"] Nov 02 18:00:01.920889 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:01.920829 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" event={"ID":"487dac52-8fbc-425e-af6c-3db7403709ff","Type":"ContainerStarted","Data":"ba3b91c304e57217e5b3aa06366ab70b5b0b7ab64154944768869ca152633235"} Nov 02 18:00:02.360724 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:02.360686 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6"] Nov 02 18:00:02.563796 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:02.563759 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv"] Nov 02 18:00:02.604107 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:02.604076 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m"] Nov 02 18:00:02.634385 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:02.634351 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr"] Nov 02 18:00:02.639434 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:02.639409 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv"] Nov 02 18:00:02.660064 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:00:02.659865 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b76941_3104_4e66_a6b0_52a20c9eb4f5.slice/crio-3feb563fbbf0a82ace7f8aa7820f11dc106a218553c6a92c5a47f89caecd8cb6 WatchSource:0}: Error finding container 3feb563fbbf0a82ace7f8aa7820f11dc106a218553c6a92c5a47f89caecd8cb6: Status 404 returned error can't find the container with id 3feb563fbbf0a82ace7f8aa7820f11dc106a218553c6a92c5a47f89caecd8cb6 Nov 02 18:00:02.677126 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:00:02.677095 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1cc711d_aed9_4bdc_84b8_795dfc52a7bd.slice/crio-e43cab02a24b61b21bbf35f2839756f5e2c7448a4da3ff19ea43476579586b8e WatchSource:0}: Error finding container e43cab02a24b61b21bbf35f2839756f5e2c7448a4da3ff19ea43476579586b8e: Status 404 returned error can't find the container with id e43cab02a24b61b21bbf35f2839756f5e2c7448a4da3ff19ea43476579586b8e Nov 02 18:00:02.677585 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:00:02.677538 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f4f507_8d94_412d_9189_7ac2a4531f01.slice/crio-60ce144a9f0a2859d4c6f04253f9b43cbad2fb3e452604f98b7629ddfc3b1bcc WatchSource:0}: Error finding container 60ce144a9f0a2859d4c6f04253f9b43cbad2fb3e452604f98b7629ddfc3b1bcc: Status 404 returned error can't find the container with id 60ce144a9f0a2859d4c6f04253f9b43cbad2fb3e452604f98b7629ddfc3b1bcc Nov 02 18:00:02.951232 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:02.951122 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" event={"ID":"65b76941-3104-4e66-a6b0-52a20c9eb4f5","Type":"ContainerStarted","Data":"3feb563fbbf0a82ace7f8aa7820f11dc106a218553c6a92c5a47f89caecd8cb6"} Nov 02 18:00:02.954032 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:02.953995 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" event={"ID":"487dac52-8fbc-425e-af6c-3db7403709ff","Type":"ContainerStarted","Data":"cbe8f3356679356541bdb55d9c18b1a97947adba3fbb0f0a27f21feff7bbda49"} Nov 02 18:00:02.962844 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:02.962811 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" event={"ID":"0ffcc622-4641-4348-ad7a-88797d8b2469","Type":"ContainerStarted","Data":"d73bc7395e47247623aa835cc0945980d024c1b1a4fbb2b5c2aab9494a8f8071"} Nov 02 18:00:02.972676 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:02.972461 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" event={"ID":"f1cc711d-aed9-4bdc-84b8-795dfc52a7bd","Type":"ContainerStarted","Data":"e43cab02a24b61b21bbf35f2839756f5e2c7448a4da3ff19ea43476579586b8e"} Nov 02 18:00:02.975762 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:02.975736 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" event={"ID":"54f4f507-8d94-412d-9189-7ac2a4531f01","Type":"ContainerStarted","Data":"60ce144a9f0a2859d4c6f04253f9b43cbad2fb3e452604f98b7629ddfc3b1bcc"} Nov 02 18:00:02.978824 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:02.978787 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" event={"ID":"0d2366ff-2ee5-4ef5-b9f8-5605a1337924","Type":"ContainerStarted","Data":"025a355128056415101f9c7aa6679564dc218ebcd0c1beec684329577334d7d2"} Nov 02 18:00:03.002411 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:03.002349 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" podStartSLOduration=3.002329948 podStartE2EDuration="3.002329948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:02.997202999 +0000 UTC m=+1821.618562776" watchObservedRunningTime="2026-11-02 18:00:03.002329948 +0000 UTC m=+1821.623689723" Nov 02 18:00:04.016444 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:04.016403 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" event={"ID":"65b76941-3104-4e66-a6b0-52a20c9eb4f5","Type":"ContainerStarted","Data":"d7b37d46cdad6785289dc06e09b7625b3a692d9b2667accbb63285452e2ee569"} Nov 02 18:00:04.018962 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:04.018933 4860 generic.go:356] "Generic (PLEG): container finished" podID="487dac52-8fbc-425e-af6c-3db7403709ff" containerID="cbe8f3356679356541bdb55d9c18b1a97947adba3fbb0f0a27f21feff7bbda49" exitCode=60 Nov 02 18:00:04.019100 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:04.018977 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" event={"ID":"487dac52-8fbc-425e-af6c-3db7403709ff","Type":"ContainerDied","Data":"cbe8f3356679356541bdb55d9c18b1a97947adba3fbb0f0a27f21feff7bbda49"} Nov 02 18:00:04.019539 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:04.019496 4860 scope.go:122] "RemoveContainer" containerID="cbe8f3356679356541bdb55d9c18b1a97947adba3fbb0f0a27f21feff7bbda49" Nov 02 18:00:04.021372 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:04.021348 4860 generic.go:356] "Generic (PLEG): container finished" podID="0ffcc622-4641-4348-ad7a-88797d8b2469" containerID="669db4b165e7a2b656f150dc3e2c71ed1f84013e1f87ec2a90e7a4cd80c7bdc2" exitCode=60 Nov 02 18:00:04.021493 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:04.021408 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" event={"ID":"0ffcc622-4641-4348-ad7a-88797d8b2469","Type":"ContainerDied","Data":"669db4b165e7a2b656f150dc3e2c71ed1f84013e1f87ec2a90e7a4cd80c7bdc2"} Nov 02 18:00:04.023427 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:04.023405 4860 scope.go:122] "RemoveContainer" containerID="669db4b165e7a2b656f150dc3e2c71ed1f84013e1f87ec2a90e7a4cd80c7bdc2" Nov 02 18:00:04.024141 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:04.024015 4860 generic.go:356] "Generic (PLEG): container finished" podID="f1cc711d-aed9-4bdc-84b8-795dfc52a7bd" containerID="4bb1bb05b29926784146e16e62c00d33e0ea2c6c08d6b1ecfaa2ca0ba6d78140" exitCode=60 Nov 02 18:00:04.024141 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:04.024098 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" event={"ID":"f1cc711d-aed9-4bdc-84b8-795dfc52a7bd","Type":"ContainerDied","Data":"4bb1bb05b29926784146e16e62c00d33e0ea2c6c08d6b1ecfaa2ca0ba6d78140"} Nov 02 18:00:04.024482 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:04.024461 4860 scope.go:122] "RemoveContainer" containerID="4bb1bb05b29926784146e16e62c00d33e0ea2c6c08d6b1ecfaa2ca0ba6d78140" Nov 02 18:00:04.027121 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:04.027098 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" event={"ID":"54f4f507-8d94-412d-9189-7ac2a4531f01","Type":"ContainerStarted","Data":"4805e7018c4fbc062bf0a440c374e41e333287e3c4fe78b800e43473ee1851f6"} Nov 02 18:00:04.030963 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:04.029491 4860 generic.go:356] "Generic (PLEG): container finished" podID="0d2366ff-2ee5-4ef5-b9f8-5605a1337924" containerID="9429f379007b8a8cc01485a8dd3d7f984ba25b3cc045f5d30621d9bcbfcc20a9" exitCode=60 Nov 02 18:00:04.030963 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:04.029596 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" event={"ID":"0d2366ff-2ee5-4ef5-b9f8-5605a1337924","Type":"ContainerDied","Data":"9429f379007b8a8cc01485a8dd3d7f984ba25b3cc045f5d30621d9bcbfcc20a9"} Nov 02 18:00:04.030963 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:04.030013 4860 scope.go:122] "RemoveContainer" containerID="9429f379007b8a8cc01485a8dd3d7f984ba25b3cc045f5d30621d9bcbfcc20a9" Nov 02 18:00:04.068424 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:04.063740 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" podStartSLOduration=4.063721101 podStartE2EDuration="4.063721101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:04.060662539 +0000 UTC m=+1822.682022318" watchObservedRunningTime="2026-11-02 18:00:04.063721101 +0000 UTC m=+1822.685080883" Nov 02 18:00:04.150369 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:04.150319 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" podStartSLOduration=4.15030353 podStartE2EDuration="4.15030353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:04.147838697 +0000 UTC m=+1822.769198471" watchObservedRunningTime="2026-11-02 18:00:04.15030353 +0000 UTC m=+1822.771663307" Nov 02 18:00:05.044241 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:05.044198 4860 generic.go:356] "Generic (PLEG): container finished" podID="65b76941-3104-4e66-a6b0-52a20c9eb4f5" containerID="d7b37d46cdad6785289dc06e09b7625b3a692d9b2667accbb63285452e2ee569" exitCode=60 Nov 02 18:00:05.044655 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:05.044286 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" event={"ID":"65b76941-3104-4e66-a6b0-52a20c9eb4f5","Type":"ContainerDied","Data":"d7b37d46cdad6785289dc06e09b7625b3a692d9b2667accbb63285452e2ee569"} Nov 02 18:00:05.044711 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:05.044686 4860 scope.go:122] "RemoveContainer" containerID="d7b37d46cdad6785289dc06e09b7625b3a692d9b2667accbb63285452e2ee569" Nov 02 18:00:05.047343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:05.046959 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" event={"ID":"487dac52-8fbc-425e-af6c-3db7403709ff","Type":"ContainerStarted","Data":"02026da472efa717be6825f495a913b0f65aa0e6f65d1febd27dfaf904673d5a"} Nov 02 18:00:05.049841 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:05.049635 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" event={"ID":"0ffcc622-4641-4348-ad7a-88797d8b2469","Type":"ContainerStarted","Data":"a22f8ff49d183fb3cdcff8d3f11ca436dbef7153d9177568362f993331ac0ae0"} Nov 02 18:00:05.052431 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:05.052406 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" event={"ID":"f1cc711d-aed9-4bdc-84b8-795dfc52a7bd","Type":"ContainerStarted","Data":"8c582108900e6d85c0e8fed04ef8a8cccff640ea405457368ebf2d30b848aa25"} Nov 02 18:00:05.056898 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:05.055998 4860 generic.go:356] "Generic (PLEG): container finished" podID="54f4f507-8d94-412d-9189-7ac2a4531f01" containerID="4805e7018c4fbc062bf0a440c374e41e333287e3c4fe78b800e43473ee1851f6" exitCode=60 Nov 02 18:00:05.056898 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:05.056117 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" event={"ID":"54f4f507-8d94-412d-9189-7ac2a4531f01","Type":"ContainerDied","Data":"4805e7018c4fbc062bf0a440c374e41e333287e3c4fe78b800e43473ee1851f6"} Nov 02 18:00:05.056898 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:05.056458 4860 scope.go:122] "RemoveContainer" containerID="4805e7018c4fbc062bf0a440c374e41e333287e3c4fe78b800e43473ee1851f6" Nov 02 18:00:05.060017 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:05.059990 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" event={"ID":"0d2366ff-2ee5-4ef5-b9f8-5605a1337924","Type":"ContainerStarted","Data":"b0271e8955f7cd145a5a5eda6177deb492a59600c9ec99de46d15b138e4eec78"} Nov 02 18:00:05.142612 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:05.142500 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" podStartSLOduration=5.14248069 podStartE2EDuration="5.14248069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:05.135023064 +0000 UTC m=+1823.756382843" watchObservedRunningTime="2026-11-02 18:00:05.14248069 +0000 UTC m=+1823.763840467" Nov 02 18:00:05.196277 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:05.196060 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" podStartSLOduration=5.196040025 podStartE2EDuration="5.196040025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:05.179185032 +0000 UTC m=+1823.800544809" watchObservedRunningTime="2026-11-02 18:00:05.196040025 +0000 UTC m=+1823.817399802" Nov 02 18:00:05.224664 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:05.224589 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" podStartSLOduration=5.224570752 podStartE2EDuration="5.224570752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:05.219709874 +0000 UTC m=+1823.841069655" watchObservedRunningTime="2026-11-02 18:00:05.224570752 +0000 UTC m=+1823.845930530" Nov 02 18:00:06.072188 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:06.072155 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" event={"ID":"65b76941-3104-4e66-a6b0-52a20c9eb4f5","Type":"ContainerStarted","Data":"074e67ce129942c1c2a055b5af9607b00e8edc228616bf3396e252734dbcc7a5"} Nov 02 18:00:06.074366 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:06.074334 4860 generic.go:356] "Generic (PLEG): container finished" podID="487dac52-8fbc-425e-af6c-3db7403709ff" containerID="02026da472efa717be6825f495a913b0f65aa0e6f65d1febd27dfaf904673d5a" exitCode=60 Nov 02 18:00:06.074484 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:06.074416 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" event={"ID":"487dac52-8fbc-425e-af6c-3db7403709ff","Type":"ContainerDied","Data":"02026da472efa717be6825f495a913b0f65aa0e6f65d1febd27dfaf904673d5a"} Nov 02 18:00:06.074484 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:06.074464 4860 scope.go:122] "RemoveContainer" containerID="cbe8f3356679356541bdb55d9c18b1a97947adba3fbb0f0a27f21feff7bbda49" Nov 02 18:00:06.074660 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:06.074638 4860 scope.go:122] "RemoveContainer" containerID="02026da472efa717be6825f495a913b0f65aa0e6f65d1febd27dfaf904673d5a" Nov 02 18:00:06.074965 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:06.074939 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h_odh-ai-gateway-infra(487dac52-8fbc-425e-af6c-3db7403709ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" podUID="487dac52-8fbc-425e-af6c-3db7403709ff" Nov 02 18:00:06.076548 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:06.076526 4860 generic.go:356] "Generic (PLEG): container finished" podID="0ffcc622-4641-4348-ad7a-88797d8b2469" containerID="a22f8ff49d183fb3cdcff8d3f11ca436dbef7153d9177568362f993331ac0ae0" exitCode=60 Nov 02 18:00:06.076644 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:06.076612 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" event={"ID":"0ffcc622-4641-4348-ad7a-88797d8b2469","Type":"ContainerDied","Data":"a22f8ff49d183fb3cdcff8d3f11ca436dbef7153d9177568362f993331ac0ae0"} Nov 02 18:00:06.076807 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:06.076790 4860 scope.go:122] "RemoveContainer" containerID="a22f8ff49d183fb3cdcff8d3f11ca436dbef7153d9177568362f993331ac0ae0" Nov 02 18:00:06.077094 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:06.077073 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6_odh-ai-gateway-infra(0ffcc622-4641-4348-ad7a-88797d8b2469)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" podUID="0ffcc622-4641-4348-ad7a-88797d8b2469" Nov 02 18:00:06.078613 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:06.078591 4860 generic.go:356] "Generic (PLEG): container finished" podID="f1cc711d-aed9-4bdc-84b8-795dfc52a7bd" containerID="8c582108900e6d85c0e8fed04ef8a8cccff640ea405457368ebf2d30b848aa25" exitCode=60 Nov 02 18:00:06.078715 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:06.078618 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" event={"ID":"f1cc711d-aed9-4bdc-84b8-795dfc52a7bd","Type":"ContainerDied","Data":"8c582108900e6d85c0e8fed04ef8a8cccff640ea405457368ebf2d30b848aa25"} Nov 02 18:00:06.078946 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:06.078924 4860 scope.go:122] "RemoveContainer" containerID="8c582108900e6d85c0e8fed04ef8a8cccff640ea405457368ebf2d30b848aa25" Nov 02 18:00:06.079189 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:06.079170 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv_odh-ai-gateway-infra(f1cc711d-aed9-4bdc-84b8-795dfc52a7bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" podUID="f1cc711d-aed9-4bdc-84b8-795dfc52a7bd" Nov 02 18:00:06.081204 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:06.081180 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" event={"ID":"54f4f507-8d94-412d-9189-7ac2a4531f01","Type":"ContainerStarted","Data":"3baecb1af5a60d93766898fa4ef9120e3e576123c5aee649fe8c86cca96333d1"} Nov 02 18:00:06.083420 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:06.083390 4860 generic.go:356] "Generic (PLEG): container finished" podID="0d2366ff-2ee5-4ef5-b9f8-5605a1337924" containerID="b0271e8955f7cd145a5a5eda6177deb492a59600c9ec99de46d15b138e4eec78" exitCode=60 Nov 02 18:00:06.083501 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:06.083452 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" event={"ID":"0d2366ff-2ee5-4ef5-b9f8-5605a1337924","Type":"ContainerDied","Data":"b0271e8955f7cd145a5a5eda6177deb492a59600c9ec99de46d15b138e4eec78"} Nov 02 18:00:06.083665 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:06.083647 4860 scope.go:122] "RemoveContainer" containerID="b0271e8955f7cd145a5a5eda6177deb492a59600c9ec99de46d15b138e4eec78" Nov 02 18:00:06.084003 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:06.083981 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m_odh-ai-gateway-infra(0d2366ff-2ee5-4ef5-b9f8-5605a1337924)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" podUID="0d2366ff-2ee5-4ef5-b9f8-5605a1337924" Nov 02 18:00:06.090274 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:06.090253 4860 scope.go:122] "RemoveContainer" containerID="669db4b165e7a2b656f150dc3e2c71ed1f84013e1f87ec2a90e7a4cd80c7bdc2" Nov 02 18:00:06.106078 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:06.106045 4860 scope.go:122] "RemoveContainer" containerID="4bb1bb05b29926784146e16e62c00d33e0ea2c6c08d6b1ecfaa2ca0ba6d78140" Nov 02 18:00:06.122197 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:06.122169 4860 scope.go:122] "RemoveContainer" containerID="9429f379007b8a8cc01485a8dd3d7f984ba25b3cc045f5d30621d9bcbfcc20a9" Nov 02 18:00:07.096021 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.095985 4860 generic.go:356] "Generic (PLEG): container finished" podID="65b76941-3104-4e66-a6b0-52a20c9eb4f5" containerID="074e67ce129942c1c2a055b5af9607b00e8edc228616bf3396e252734dbcc7a5" exitCode=60 Nov 02 18:00:07.096480 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.096062 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" event={"ID":"65b76941-3104-4e66-a6b0-52a20c9eb4f5","Type":"ContainerDied","Data":"074e67ce129942c1c2a055b5af9607b00e8edc228616bf3396e252734dbcc7a5"} Nov 02 18:00:07.096480 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.096125 4860 scope.go:122] "RemoveContainer" containerID="d7b37d46cdad6785289dc06e09b7625b3a692d9b2667accbb63285452e2ee569" Nov 02 18:00:07.096480 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.096452 4860 scope.go:122] "RemoveContainer" containerID="074e67ce129942c1c2a055b5af9607b00e8edc228616bf3396e252734dbcc7a5" Nov 02 18:00:07.096943 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:07.096920 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv_odh-ai-gateway-infra(65b76941-3104-4e66-a6b0-52a20c9eb4f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" podUID="65b76941-3104-4e66-a6b0-52a20c9eb4f5" Nov 02 18:00:07.098383 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.098361 4860 scope.go:122] "RemoveContainer" containerID="02026da472efa717be6825f495a913b0f65aa0e6f65d1febd27dfaf904673d5a" Nov 02 18:00:07.098772 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:07.098751 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h_odh-ai-gateway-infra(487dac52-8fbc-425e-af6c-3db7403709ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" podUID="487dac52-8fbc-425e-af6c-3db7403709ff" Nov 02 18:00:07.100190 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.100173 4860 scope.go:122] "RemoveContainer" containerID="a22f8ff49d183fb3cdcff8d3f11ca436dbef7153d9177568362f993331ac0ae0" Nov 02 18:00:07.100550 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:07.100521 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6_odh-ai-gateway-infra(0ffcc622-4641-4348-ad7a-88797d8b2469)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" podUID="0ffcc622-4641-4348-ad7a-88797d8b2469" Nov 02 18:00:07.102026 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.102001 4860 scope.go:122] "RemoveContainer" containerID="8c582108900e6d85c0e8fed04ef8a8cccff640ea405457368ebf2d30b848aa25" Nov 02 18:00:07.102378 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:07.102354 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv_odh-ai-gateway-infra(f1cc711d-aed9-4bdc-84b8-795dfc52a7bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" podUID="f1cc711d-aed9-4bdc-84b8-795dfc52a7bd" Nov 02 18:00:07.103665 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.103645 4860 generic.go:356] "Generic (PLEG): container finished" podID="54f4f507-8d94-412d-9189-7ac2a4531f01" containerID="3baecb1af5a60d93766898fa4ef9120e3e576123c5aee649fe8c86cca96333d1" exitCode=60 Nov 02 18:00:07.103769 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.103733 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" event={"ID":"54f4f507-8d94-412d-9189-7ac2a4531f01","Type":"ContainerDied","Data":"3baecb1af5a60d93766898fa4ef9120e3e576123c5aee649fe8c86cca96333d1"} Nov 02 18:00:07.103962 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.103945 4860 scope.go:122] "RemoveContainer" containerID="3baecb1af5a60d93766898fa4ef9120e3e576123c5aee649fe8c86cca96333d1" Nov 02 18:00:07.104262 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:07.104237 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr_odh-ai-gateway-infra(54f4f507-8d94-412d-9189-7ac2a4531f01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" podUID="54f4f507-8d94-412d-9189-7ac2a4531f01" Nov 02 18:00:07.105887 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.105868 4860 scope.go:122] "RemoveContainer" containerID="b0271e8955f7cd145a5a5eda6177deb492a59600c9ec99de46d15b138e4eec78" Nov 02 18:00:07.106127 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:07.106110 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m_odh-ai-gateway-infra(0d2366ff-2ee5-4ef5-b9f8-5605a1337924)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" podUID="0d2366ff-2ee5-4ef5-b9f8-5605a1337924" Nov 02 18:00:07.111622 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.111604 4860 scope.go:122] "RemoveContainer" containerID="4805e7018c4fbc062bf0a440c374e41e333287e3c4fe78b800e43473ee1851f6" Nov 02 18:00:07.344472 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.344421 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68"] Nov 02 18:00:07.351085 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.351012 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.374521 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.374492 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68"] Nov 02 18:00:07.452252 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.452218 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.452464 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.452359 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.452464 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.452454 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.452583 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.452507 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.452583 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.452531 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.452583 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.452554 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.452735 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.452695 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a03fdb-9060-4f2c-a96e-d78764568df8-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.452810 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.452793 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grtgg\" (UniqueName: \"kubernetes.io/projected/39a03fdb-9060-4f2c-a96e-d78764568df8-kube-api-access-grtgg\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.452924 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.452911 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.554742 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.554705 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.554965 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.554756 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.554965 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.554807 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.554965 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.554830 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.554965 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.554867 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.555169 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.555005 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a03fdb-9060-4f2c-a96e-d78764568df8-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.555169 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.555149 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grtgg\" (UniqueName: \"kubernetes.io/projected/39a03fdb-9060-4f2c-a96e-d78764568df8-kube-api-access-grtgg\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.555294 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.555275 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.555378 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.555282 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.555378 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.555307 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.555510 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.555476 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.555567 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.555508 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.555769 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.555742 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.555863 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.555824 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a03fdb-9060-4f2c-a96e-d78764568df8-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.557586 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.557558 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.557714 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.557699 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.581077 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.581047 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.581925 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.581903 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grtgg\" (UniqueName: \"kubernetes.io/projected/39a03fdb-9060-4f2c-a96e-d78764568df8-kube-api-access-grtgg\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.668375 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.668302 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:07.998294 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:00:07.998229 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a03fdb_9060_4f2c_a96e_d78764568df8.slice/crio-c30ea041248743c96328ffe2871eb9eb8562c58803e0929c32da6cb7fa6a1c15 WatchSource:0}: Error finding container c30ea041248743c96328ffe2871eb9eb8562c58803e0929c32da6cb7fa6a1c15: Status 404 returned error can't find the container with id c30ea041248743c96328ffe2871eb9eb8562c58803e0929c32da6cb7fa6a1c15 Nov 02 18:00:07.998937 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:07.998912 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68"] Nov 02 18:00:08.001704 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:08.001668 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:00:08.001801 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:08.001734 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:00:08.001801 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:08.001783 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:00:08.119442 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:08.119396 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" event={"ID":"39a03fdb-9060-4f2c-a96e-d78764568df8","Type":"ContainerStarted","Data":"ed501750397a99495279290d86a0add1ae10c39be1e6acacffde3bbf5022d625"} Nov 02 18:00:08.119931 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:08.119473 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" event={"ID":"39a03fdb-9060-4f2c-a96e-d78764568df8","Type":"ContainerStarted","Data":"c30ea041248743c96328ffe2871eb9eb8562c58803e0929c32da6cb7fa6a1c15"} Nov 02 18:00:08.121997 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:08.121965 4860 scope.go:122] "RemoveContainer" containerID="074e67ce129942c1c2a055b5af9607b00e8edc228616bf3396e252734dbcc7a5" Nov 02 18:00:08.122412 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:08.122384 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv_odh-ai-gateway-infra(65b76941-3104-4e66-a6b0-52a20c9eb4f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" podUID="65b76941-3104-4e66-a6b0-52a20c9eb4f5" Nov 02 18:00:08.125861 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:08.125834 4860 scope.go:122] "RemoveContainer" containerID="3baecb1af5a60d93766898fa4ef9120e3e576123c5aee649fe8c86cca96333d1" Nov 02 18:00:08.126256 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:08.126234 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr_odh-ai-gateway-infra(54f4f507-8d94-412d-9189-7ac2a4531f01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" podUID="54f4f507-8d94-412d-9189-7ac2a4531f01" Nov 02 18:00:08.171633 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:08.171569 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" podStartSLOduration=1.171555182 podStartE2EDuration="1.171555182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:08.165203154 +0000 UTC m=+1826.786562930" watchObservedRunningTime="2026-11-02 18:00:08.171555182 +0000 UTC m=+1826.792914957" Nov 02 18:00:08.671915 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:08.671870 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:09.678566 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:09.678526 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 18:00:09.679164 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:09.678950 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" podUID="39a03fdb-9060-4f2c-a96e-d78764568df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 18:00:10.678980 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:10.673467 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 18:00:10.678980 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:10.673543 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" podUID="39a03fdb-9060-4f2c-a96e-d78764568df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 18:00:11.677875 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:11.677277 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 18:00:11.677875 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:11.677343 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" podUID="39a03fdb-9060-4f2c-a96e-d78764568df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 18:00:12.677272 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:12.677235 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 18:00:12.677898 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:12.677863 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" podUID="39a03fdb-9060-4f2c-a96e-d78764568df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 18:00:13.679874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:13.671402 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 18:00:13.679874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:13.671492 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" podUID="39a03fdb-9060-4f2c-a96e-d78764568df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 18:00:14.673834 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:14.673795 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:14.674139 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:14.674118 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:14.674976 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:14.674948 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:00:16.491826 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:16.491793 4860 scope.go:122] "RemoveContainer" containerID="3baecb1af5a60d93766898fa4ef9120e3e576123c5aee649fe8c86cca96333d1" Nov 02 18:00:17.232436 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.232399 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd"] Nov 02 18:00:17.237461 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.237434 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" Nov 02 18:00:17.242053 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.242015 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" event={"ID":"54f4f507-8d94-412d-9189-7ac2a4531f01","Type":"ContainerStarted","Data":"5fb05b2331da902af11f6a9a9c4b08bd729692bfdced974edd101dcc186b0f8d"} Nov 02 18:00:17.263707 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.263676 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5fd3e0-serving-cert\"" Nov 02 18:00:17.286295 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.286257 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd"] Nov 02 18:00:17.384437 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.384399 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zprkb\" (UniqueName: \"kubernetes.io/projected/ceba04e6-9968-4301-aaa3-6372edea5720-kube-api-access-zprkb\") pod \"maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd\" (UID: \"ceba04e6-9968-4301-aaa3-6372edea5720\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" Nov 02 18:00:17.384728 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.384704 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceba04e6-9968-4301-aaa3-6372edea5720-metrics-certs\") pod \"maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd\" (UID: \"ceba04e6-9968-4301-aaa3-6372edea5720\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" Nov 02 18:00:17.384953 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.384926 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceba04e6-9968-4301-aaa3-6372edea5720-maas-api-tls\") pod \"maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd\" (UID: \"ceba04e6-9968-4301-aaa3-6372edea5720\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" Nov 02 18:00:17.385053 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.385039 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceba04e6-9968-4301-aaa3-6372edea5720-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd\" (UID: \"ceba04e6-9968-4301-aaa3-6372edea5720\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" Nov 02 18:00:17.488467 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.488370 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceba04e6-9968-4301-aaa3-6372edea5720-maas-api-tls\") pod \"maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd\" (UID: \"ceba04e6-9968-4301-aaa3-6372edea5720\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" Nov 02 18:00:17.489479 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.489453 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceba04e6-9968-4301-aaa3-6372edea5720-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd\" (UID: \"ceba04e6-9968-4301-aaa3-6372edea5720\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" Nov 02 18:00:17.490165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.490144 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zprkb\" (UniqueName: \"kubernetes.io/projected/ceba04e6-9968-4301-aaa3-6372edea5720-kube-api-access-zprkb\") pod \"maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd\" (UID: \"ceba04e6-9968-4301-aaa3-6372edea5720\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" Nov 02 18:00:17.500030 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.491736 4860 scope.go:122] "RemoveContainer" containerID="a22f8ff49d183fb3cdcff8d3f11ca436dbef7153d9177568362f993331ac0ae0" Nov 02 18:00:17.500458 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.500134 4860 scope.go:122] "RemoveContainer" containerID="b0271e8955f7cd145a5a5eda6177deb492a59600c9ec99de46d15b138e4eec78" Nov 02 18:00:17.501724 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.501315 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceba04e6-9968-4301-aaa3-6372edea5720-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd\" (UID: \"ceba04e6-9968-4301-aaa3-6372edea5720\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" Nov 02 18:00:17.501724 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.501478 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceba04e6-9968-4301-aaa3-6372edea5720-metrics-certs\") pod \"maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd\" (UID: \"ceba04e6-9968-4301-aaa3-6372edea5720\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" Nov 02 18:00:17.504464 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.504433 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceba04e6-9968-4301-aaa3-6372edea5720-maas-api-tls\") pod \"maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd\" (UID: \"ceba04e6-9968-4301-aaa3-6372edea5720\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" Nov 02 18:00:17.505227 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.505208 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceba04e6-9968-4301-aaa3-6372edea5720-metrics-certs\") pod \"maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd\" (UID: \"ceba04e6-9968-4301-aaa3-6372edea5720\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" Nov 02 18:00:17.517601 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.517566 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zprkb\" (UniqueName: \"kubernetes.io/projected/ceba04e6-9968-4301-aaa3-6372edea5720-kube-api-access-zprkb\") pod \"maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd\" (UID: \"ceba04e6-9968-4301-aaa3-6372edea5720\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" Nov 02 18:00:17.573737 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.573700 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" Nov 02 18:00:17.907510 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:00:17.907476 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceba04e6_9968_4301_aaa3_6372edea5720.slice/crio-256a0131e18b82d9b1c7ed932741bcbb48de501c2f67ddd246ccbb7597f5b0df WatchSource:0}: Error finding container 256a0131e18b82d9b1c7ed932741bcbb48de501c2f67ddd246ccbb7597f5b0df: Status 404 returned error can't find the container with id 256a0131e18b82d9b1c7ed932741bcbb48de501c2f67ddd246ccbb7597f5b0df Nov 02 18:00:17.907955 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:17.907934 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd"] Nov 02 18:00:18.254155 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:18.254106 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" event={"ID":"ceba04e6-9968-4301-aaa3-6372edea5720","Type":"ContainerStarted","Data":"256a0131e18b82d9b1c7ed932741bcbb48de501c2f67ddd246ccbb7597f5b0df"} Nov 02 18:00:18.256574 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:18.256534 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" event={"ID":"0ffcc622-4641-4348-ad7a-88797d8b2469","Type":"ContainerStarted","Data":"f0d5e208abef24b4ca3589590f8fb06e4a8a05d2f6c8776a6af45318769f9591"} Nov 02 18:00:18.260292 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:18.260267 4860 generic.go:356] "Generic (PLEG): container finished" podID="54f4f507-8d94-412d-9189-7ac2a4531f01" containerID="5fb05b2331da902af11f6a9a9c4b08bd729692bfdced974edd101dcc186b0f8d" exitCode=60 Nov 02 18:00:18.260409 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:18.260344 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" event={"ID":"54f4f507-8d94-412d-9189-7ac2a4531f01","Type":"ContainerDied","Data":"5fb05b2331da902af11f6a9a9c4b08bd729692bfdced974edd101dcc186b0f8d"} Nov 02 18:00:18.260484 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:18.260394 4860 scope.go:122] "RemoveContainer" containerID="3baecb1af5a60d93766898fa4ef9120e3e576123c5aee649fe8c86cca96333d1" Nov 02 18:00:18.260824 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:18.260804 4860 scope.go:122] "RemoveContainer" containerID="5fb05b2331da902af11f6a9a9c4b08bd729692bfdced974edd101dcc186b0f8d" Nov 02 18:00:18.261243 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:18.261210 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr_odh-ai-gateway-infra(54f4f507-8d94-412d-9189-7ac2a4531f01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" podUID="54f4f507-8d94-412d-9189-7ac2a4531f01" Nov 02 18:00:18.263804 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:18.263781 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" event={"ID":"0d2366ff-2ee5-4ef5-b9f8-5605a1337924","Type":"ContainerStarted","Data":"d2b658deb3fd3f5a7270d167859020e8ac1f955c1d9149f7297762405dafd5ee"} Nov 02 18:00:18.423516 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:18.419899 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr"] Nov 02 18:00:18.501589 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:18.493604 4860 scope.go:122] "RemoveContainer" containerID="8c582108900e6d85c0e8fed04ef8a8cccff640ea405457368ebf2d30b848aa25" Nov 02 18:00:19.277205 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.277163 4860 generic.go:356] "Generic (PLEG): container finished" podID="0ffcc622-4641-4348-ad7a-88797d8b2469" containerID="f0d5e208abef24b4ca3589590f8fb06e4a8a05d2f6c8776a6af45318769f9591" exitCode=60 Nov 02 18:00:19.277369 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.277314 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" event={"ID":"0ffcc622-4641-4348-ad7a-88797d8b2469","Type":"ContainerDied","Data":"f0d5e208abef24b4ca3589590f8fb06e4a8a05d2f6c8776a6af45318769f9591"} Nov 02 18:00:19.277416 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.277374 4860 scope.go:122] "RemoveContainer" containerID="a22f8ff49d183fb3cdcff8d3f11ca436dbef7153d9177568362f993331ac0ae0" Nov 02 18:00:19.278873 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.278809 4860 scope.go:122] "RemoveContainer" containerID="f0d5e208abef24b4ca3589590f8fb06e4a8a05d2f6c8776a6af45318769f9591" Nov 02 18:00:19.279285 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:19.279253 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6_odh-ai-gateway-infra(0ffcc622-4641-4348-ad7a-88797d8b2469)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" podUID="0ffcc622-4641-4348-ad7a-88797d8b2469" Nov 02 18:00:19.284999 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.284968 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" event={"ID":"f1cc711d-aed9-4bdc-84b8-795dfc52a7bd","Type":"ContainerStarted","Data":"1f21a8826a17bfe12298a35acdc08a92ed4531f56f1e2f111db573722f4971bd"} Nov 02 18:00:19.291751 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.291018 4860 generic.go:356] "Generic (PLEG): container finished" podID="0d2366ff-2ee5-4ef5-b9f8-5605a1337924" containerID="d2b658deb3fd3f5a7270d167859020e8ac1f955c1d9149f7297762405dafd5ee" exitCode=60 Nov 02 18:00:19.291751 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.291245 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" event={"ID":"0d2366ff-2ee5-4ef5-b9f8-5605a1337924","Type":"ContainerDied","Data":"d2b658deb3fd3f5a7270d167859020e8ac1f955c1d9149f7297762405dafd5ee"} Nov 02 18:00:19.291751 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.291747 4860 scope.go:122] "RemoveContainer" containerID="d2b658deb3fd3f5a7270d167859020e8ac1f955c1d9149f7297762405dafd5ee" Nov 02 18:00:19.292384 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:19.292354 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m_odh-ai-gateway-infra(0d2366ff-2ee5-4ef5-b9f8-5605a1337924)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" podUID="0d2366ff-2ee5-4ef5-b9f8-5605a1337924" Nov 02 18:00:19.294755 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.294730 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" event={"ID":"ceba04e6-9968-4301-aaa3-6372edea5720","Type":"ContainerStarted","Data":"337bf193f9d841beae50b0763ffc4658f22c84c3df103a4610fe9573c8eed447"} Nov 02 18:00:19.295486 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.295320 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" Nov 02 18:00:19.306579 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.306553 4860 scope.go:122] "RemoveContainer" containerID="b0271e8955f7cd145a5a5eda6177deb492a59600c9ec99de46d15b138e4eec78" Nov 02 18:00:19.320176 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.320112 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6"] Nov 02 18:00:19.405471 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.405435 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m"] Nov 02 18:00:19.430808 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.430753 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" podStartSLOduration=2.005227454 podStartE2EDuration="2.430739964s" totalImagesPullingTime="425.51251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:19.422024886 +0000 UTC m=+1838.043384664" watchObservedRunningTime="2026-11-02 18:00:19.430739964 +0000 UTC m=+1838.052099771" Nov 02 18:00:19.483610 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.483586 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" Nov 02 18:00:19.491559 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.491538 4860 scope.go:122] "RemoveContainer" containerID="02026da472efa717be6825f495a913b0f65aa0e6f65d1febd27dfaf904673d5a" Nov 02 18:00:19.491753 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.491731 4860 scope.go:122] "RemoveContainer" containerID="074e67ce129942c1c2a055b5af9607b00e8edc228616bf3396e252734dbcc7a5" Nov 02 18:00:19.586243 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.586175 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54f4f507-8d94-412d-9189-7ac2a4531f01-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f4f507-8d94-412d-9189-7ac2a4531f01-maas-api-service-ca\") pod \"54f4f507-8d94-412d-9189-7ac2a4531f01\" (UID: \"54f4f507-8d94-412d-9189-7ac2a4531f01\") " Nov 02 18:00:19.587345 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.587320 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f4f507-8d94-412d-9189-7ac2a4531f01-kube-api-access-pk6h5\" (UniqueName: \"kubernetes.io/projected/54f4f507-8d94-412d-9189-7ac2a4531f01-kube-api-access-pk6h5\") pod \"54f4f507-8d94-412d-9189-7ac2a4531f01\" (UID: \"54f4f507-8d94-412d-9189-7ac2a4531f01\") " Nov 02 18:00:19.588081 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.587103 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f4f507-8d94-412d-9189-7ac2a4531f01-maas-api-service-ca" pod "54f4f507-8d94-412d-9189-7ac2a4531f01" (UID: "54f4f507-8d94-412d-9189-7ac2a4531f01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:19.589197 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.589175 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f4f507-8d94-412d-9189-7ac2a4531f01-maas-api-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:19.591199 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.591173 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f4f507-8d94-412d-9189-7ac2a4531f01-kube-api-access-pk6h5" pod "54f4f507-8d94-412d-9189-7ac2a4531f01" (UID: "54f4f507-8d94-412d-9189-7ac2a4531f01"). InnerVolumeSpecName "kube-api-access-pk6h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:19.691819 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:19.691783 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk6h5\" (UniqueName: \"kubernetes.io/projected/54f4f507-8d94-412d-9189-7ac2a4531f01-kube-api-access-pk6h5\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:20.307376 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.307344 4860 generic.go:356] "Generic (PLEG): container finished" podID="f1cc711d-aed9-4bdc-84b8-795dfc52a7bd" containerID="1f21a8826a17bfe12298a35acdc08a92ed4531f56f1e2f111db573722f4971bd" exitCode=60 Nov 02 18:00:20.307541 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.307415 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" event={"ID":"f1cc711d-aed9-4bdc-84b8-795dfc52a7bd","Type":"ContainerDied","Data":"1f21a8826a17bfe12298a35acdc08a92ed4531f56f1e2f111db573722f4971bd"} Nov 02 18:00:20.307541 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.307486 4860 scope.go:122] "RemoveContainer" containerID="8c582108900e6d85c0e8fed04ef8a8cccff640ea405457368ebf2d30b848aa25" Nov 02 18:00:20.307900 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.307878 4860 scope.go:122] "RemoveContainer" containerID="1f21a8826a17bfe12298a35acdc08a92ed4531f56f1e2f111db573722f4971bd" Nov 02 18:00:20.308282 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:20.308255 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv_odh-ai-gateway-infra(f1cc711d-aed9-4bdc-84b8-795dfc52a7bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" podUID="f1cc711d-aed9-4bdc-84b8-795dfc52a7bd" Nov 02 18:00:20.309062 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.309044 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" Nov 02 18:00:20.309062 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.309048 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr" event={"ID":"54f4f507-8d94-412d-9189-7ac2a4531f01","Type":"ContainerDied","Data":"60ce144a9f0a2859d4c6f04253f9b43cbad2fb3e452604f98b7629ddfc3b1bcc"} Nov 02 18:00:20.313720 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.313692 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" event={"ID":"65b76941-3104-4e66-a6b0-52a20c9eb4f5","Type":"ContainerStarted","Data":"f620515b015815da21213c282dd8605f0d9e511d4c97cc3bc27fdfd39375cb04"} Nov 02 18:00:20.316437 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.316413 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" event={"ID":"487dac52-8fbc-425e-af6c-3db7403709ff","Type":"ContainerStarted","Data":"624de86b97fca222861cf1ae09c22072bed1b6acc1edef899866b704a81a535f"} Nov 02 18:00:20.324329 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.324310 4860 scope.go:122] "RemoveContainer" containerID="5fb05b2331da902af11f6a9a9c4b08bd729692bfdced974edd101dcc186b0f8d" Nov 02 18:00:20.483749 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.483652 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr"] Nov 02 18:00:20.489228 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.489198 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a29999-29894040-bf2jr"] Nov 02 18:00:20.524015 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.523977 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" Nov 02 18:00:20.524434 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.524410 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv"] Nov 02 18:00:20.587150 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.587075 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" Nov 02 18:00:20.596439 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.596409 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr"] Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.597940 4860 cpu_manager.go:414] "containerMap: removing container" podUID="0ffcc622-4641-4348-ad7a-88797d8b2469" containerName="cleanup" Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.597967 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ffcc622-4641-4348-ad7a-88797d8b2469" containerName="cleanup" Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.597984 4860 cpu_manager.go:414] "containerMap: removing container" podUID="0ffcc622-4641-4348-ad7a-88797d8b2469" containerName="cleanup" Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.597994 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ffcc622-4641-4348-ad7a-88797d8b2469" containerName="cleanup" Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.598033 4860 cpu_manager.go:414] "containerMap: removing container" podUID="54f4f507-8d94-412d-9189-7ac2a4531f01" containerName="cleanup" Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.598041 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f4f507-8d94-412d-9189-7ac2a4531f01" containerName="cleanup" Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.598053 4860 cpu_manager.go:414] "containerMap: removing container" podUID="0d2366ff-2ee5-4ef5-b9f8-5605a1337924" containerName="cleanup" Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.598060 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2366ff-2ee5-4ef5-b9f8-5605a1337924" containerName="cleanup" Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.598071 4860 cpu_manager.go:414] "containerMap: removing container" podUID="0ffcc622-4641-4348-ad7a-88797d8b2469" containerName="cleanup" Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.598078 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ffcc622-4641-4348-ad7a-88797d8b2469" containerName="cleanup" Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.598105 4860 cpu_manager.go:414] "containerMap: removing container" podUID="54f4f507-8d94-412d-9189-7ac2a4531f01" containerName="cleanup" Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.598114 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f4f507-8d94-412d-9189-7ac2a4531f01" containerName="cleanup" Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.598187 4860 cpu_manager.go:414] "containerMap: removing container" podUID="0d2366ff-2ee5-4ef5-b9f8-5605a1337924" containerName="cleanup" Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.598196 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2366ff-2ee5-4ef5-b9f8-5605a1337924" containerName="cleanup" Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.598381 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f4f507-8d94-412d-9189-7ac2a4531f01" containerName="cleanup" Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.598396 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2366ff-2ee5-4ef5-b9f8-5605a1337924" containerName="cleanup" Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.598408 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ffcc622-4641-4348-ad7a-88797d8b2469" containerName="cleanup" Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.598422 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f4f507-8d94-412d-9189-7ac2a4531f01" containerName="cleanup" Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.598443 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2366ff-2ee5-4ef5-b9f8-5605a1337924" containerName="cleanup" Nov 02 18:00:20.600528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.598456 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ffcc622-4641-4348-ad7a-88797d8b2469" containerName="cleanup" Nov 02 18:00:20.602480 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.602457 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.626091 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.625767 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-872354-openshift-default-dockercfg-rdxd7\"" Nov 02 18:00:20.626225 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.626200 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ffcc622-4641-4348-ad7a-88797d8b2469-kube-api-access-tmbnh\" (UniqueName: \"kubernetes.io/projected/0ffcc622-4641-4348-ad7a-88797d8b2469-kube-api-access-tmbnh\") pod \"0ffcc622-4641-4348-ad7a-88797d8b2469\" (UID: \"0ffcc622-4641-4348-ad7a-88797d8b2469\") " Nov 02 18:00:20.626372 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.626346 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ffcc622-4641-4348-ad7a-88797d8b2469-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ffcc622-4641-4348-ad7a-88797d8b2469-maas-api-service-ca\") pod \"0ffcc622-4641-4348-ad7a-88797d8b2469\" (UID: \"0ffcc622-4641-4348-ad7a-88797d8b2469\") " Nov 02 18:00:20.628268 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.628213 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ffcc622-4641-4348-ad7a-88797d8b2469-maas-api-service-ca" pod "0ffcc622-4641-4348-ad7a-88797d8b2469" (UID: "0ffcc622-4641-4348-ad7a-88797d8b2469"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:20.636869 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.636828 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr"] Nov 02 18:00:20.642367 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.642336 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ffcc622-4641-4348-ad7a-88797d8b2469-kube-api-access-tmbnh" pod "0ffcc622-4641-4348-ad7a-88797d8b2469" (UID: "0ffcc622-4641-4348-ad7a-88797d8b2469"). InnerVolumeSpecName "kube-api-access-tmbnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:20.727871 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.727818 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2366ff-2ee5-4ef5-b9f8-5605a1337924-kube-api-access-djs6b\" (UniqueName: \"kubernetes.io/projected/0d2366ff-2ee5-4ef5-b9f8-5605a1337924-kube-api-access-djs6b\") pod \"0d2366ff-2ee5-4ef5-b9f8-5605a1337924\" (UID: \"0d2366ff-2ee5-4ef5-b9f8-5605a1337924\") " Nov 02 18:00:20.728317 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.728294 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d2366ff-2ee5-4ef5-b9f8-5605a1337924-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2366ff-2ee5-4ef5-b9f8-5605a1337924-maas-api-service-ca\") pod \"0d2366ff-2ee5-4ef5-b9f8-5605a1337924\" (UID: \"0d2366ff-2ee5-4ef5-b9f8-5605a1337924\") " Nov 02 18:00:20.728662 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.728641 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-workload-certs\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.728733 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.728654 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2366ff-2ee5-4ef5-b9f8-5605a1337924-maas-api-service-ca" pod "0d2366ff-2ee5-4ef5-b9f8-5605a1337924" (UID: "0d2366ff-2ee5-4ef5-b9f8-5605a1337924"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:20.728788 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.728760 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbspf\" (UniqueName: \"kubernetes.io/projected/5b585147-38f8-4a8f-be0b-d1ffaef9481d-kube-api-access-hbspf\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.728841 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.728821 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-credential-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.728932 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.728870 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-data\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.728989 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.728932 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-workload-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.729044 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.728999 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istiod-ca-cert\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.729094 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.729078 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-podinfo\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.729144 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.729122 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-envoy\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.729364 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.729343 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-token\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.729545 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.729525 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2366ff-2ee5-4ef5-b9f8-5605a1337924-maas-api-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:20.729604 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.729553 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmbnh\" (UniqueName: \"kubernetes.io/projected/0ffcc622-4641-4348-ad7a-88797d8b2469-kube-api-access-tmbnh\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:20.729604 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.729566 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ffcc622-4641-4348-ad7a-88797d8b2469-maas-api-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:20.730390 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.730366 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2366ff-2ee5-4ef5-b9f8-5605a1337924-kube-api-access-djs6b" pod "0d2366ff-2ee5-4ef5-b9f8-5605a1337924" (UID: "0d2366ff-2ee5-4ef5-b9f8-5605a1337924"). InnerVolumeSpecName "kube-api-access-djs6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:20.831310 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.831272 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-token\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.831503 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.831457 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-workload-certs\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.831574 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.831513 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbspf\" (UniqueName: \"kubernetes.io/projected/5b585147-38f8-4a8f-be0b-d1ffaef9481d-kube-api-access-hbspf\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.831574 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.831563 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-credential-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.831674 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.831589 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-data\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.831674 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.831644 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-workload-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.831810 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.831704 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istiod-ca-cert\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.831810 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.831779 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-podinfo\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.831997 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.831821 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-envoy\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.831997 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.831919 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-workload-certs\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.832161 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.832135 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-data\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.832231 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.832164 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djs6b\" (UniqueName: \"kubernetes.io/projected/0d2366ff-2ee5-4ef5-b9f8-5605a1337924-kube-api-access-djs6b\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:20.832231 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.832183 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-workload-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.832379 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.832349 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-credential-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.832929 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.832901 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istiod-ca-cert\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.834435 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.834400 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-envoy\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.835095 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.835045 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-podinfo\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.855434 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.855382 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-token\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.855694 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.855672 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbspf\" (UniqueName: \"kubernetes.io/projected/5b585147-38f8-4a8f-be0b-d1ffaef9481d-kube-api-access-hbspf\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jrbkr\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:20.919755 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:20.919724 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:21.246424 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:00:21.246393 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b585147_38f8_4a8f_be0b_d1ffaef9481d.slice/crio-ad1c66cc7249e727abaa2052b649ebd5620a279e49f13ee68e848b988e72c2b6 WatchSource:0}: Error finding container ad1c66cc7249e727abaa2052b649ebd5620a279e49f13ee68e848b988e72c2b6: Status 404 returned error can't find the container with id ad1c66cc7249e727abaa2052b649ebd5620a279e49f13ee68e848b988e72c2b6 Nov 02 18:00:21.248287 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.248193 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr"] Nov 02 18:00:21.249564 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.249532 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:00:21.249632 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.249597 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:00:21.249632 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.249625 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:00:21.328065 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.328039 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" Nov 02 18:00:21.328231 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.328075 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m" event={"ID":"0d2366ff-2ee5-4ef5-b9f8-5605a1337924","Type":"ContainerDied","Data":"025a355128056415101f9c7aa6679564dc218ebcd0c1beec684329577334d7d2"} Nov 02 18:00:21.328231 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.328149 4860 scope.go:122] "RemoveContainer" containerID="d2b658deb3fd3f5a7270d167859020e8ac1f955c1d9149f7297762405dafd5ee" Nov 02 18:00:21.334896 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.332206 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" event={"ID":"5b585147-38f8-4a8f-be0b-d1ffaef9481d","Type":"ContainerStarted","Data":"0347bdf1fd1f77a0646b48b520e91b4a1a7cb0a9dcd686ffc1611e359031d770"} Nov 02 18:00:21.334896 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.332262 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" event={"ID":"5b585147-38f8-4a8f-be0b-d1ffaef9481d","Type":"ContainerStarted","Data":"ad1c66cc7249e727abaa2052b649ebd5620a279e49f13ee68e848b988e72c2b6"} Nov 02 18:00:21.335103 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.335080 4860 generic.go:356] "Generic (PLEG): container finished" podID="65b76941-3104-4e66-a6b0-52a20c9eb4f5" containerID="f620515b015815da21213c282dd8605f0d9e511d4c97cc3bc27fdfd39375cb04" exitCode=60 Nov 02 18:00:21.335188 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.335159 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" event={"ID":"65b76941-3104-4e66-a6b0-52a20c9eb4f5","Type":"ContainerDied","Data":"f620515b015815da21213c282dd8605f0d9e511d4c97cc3bc27fdfd39375cb04"} Nov 02 18:00:21.335657 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.335635 4860 scope.go:122] "RemoveContainer" containerID="f620515b015815da21213c282dd8605f0d9e511d4c97cc3bc27fdfd39375cb04" Nov 02 18:00:21.336148 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:21.336122 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv_odh-ai-gateway-infra(65b76941-3104-4e66-a6b0-52a20c9eb4f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" podUID="65b76941-3104-4e66-a6b0-52a20c9eb4f5" Nov 02 18:00:21.337512 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.337489 4860 generic.go:356] "Generic (PLEG): container finished" podID="487dac52-8fbc-425e-af6c-3db7403709ff" containerID="624de86b97fca222861cf1ae09c22072bed1b6acc1edef899866b704a81a535f" exitCode=60 Nov 02 18:00:21.337616 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.337534 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" event={"ID":"487dac52-8fbc-425e-af6c-3db7403709ff","Type":"ContainerDied","Data":"624de86b97fca222861cf1ae09c22072bed1b6acc1edef899866b704a81a535f"} Nov 02 18:00:21.338143 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.338008 4860 scope.go:122] "RemoveContainer" containerID="624de86b97fca222861cf1ae09c22072bed1b6acc1edef899866b704a81a535f" Nov 02 18:00:21.338411 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:21.338385 4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h_odh-ai-gateway-infra(487dac52-8fbc-425e-af6c-3db7403709ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" podUID="487dac52-8fbc-425e-af6c-3db7403709ff" Nov 02 18:00:21.339493 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.339474 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" Nov 02 18:00:21.339738 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.339503 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6" event={"ID":"0ffcc622-4641-4348-ad7a-88797d8b2469","Type":"ContainerDied","Data":"d73bc7395e47247623aa835cc0945980d024c1b1a4fbb2b5c2aab9494a8f8071"} Nov 02 18:00:21.351597 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.351571 4860 scope.go:122] "RemoveContainer" containerID="074e67ce129942c1c2a055b5af9607b00e8edc228616bf3396e252734dbcc7a5" Nov 02 18:00:21.371535 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.371505 4860 scope.go:122] "RemoveContainer" containerID="02026da472efa717be6825f495a913b0f65aa0e6f65d1febd27dfaf904673d5a" Nov 02 18:00:21.396323 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.395499 4860 scope.go:122] "RemoveContainer" containerID="f0d5e208abef24b4ca3589590f8fb06e4a8a05d2f6c8776a6af45318769f9591" Nov 02 18:00:21.400168 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.396930 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m"] Nov 02 18:00:21.405772 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.405742 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2c9b1d-29894040-c466m"] Nov 02 18:00:21.472214 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.470521 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" podStartSLOduration=1.470501931 podStartE2EDuration="1.470501931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:21.447668498 +0000 UTC m=+1840.069028275" watchObservedRunningTime="2026-11-02 18:00:21.470501931 +0000 UTC m=+1840.091861709" Nov 02 18:00:21.510119 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.510051 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d2366ff-2ee5-4ef5-b9f8-5605a1337924" path="/var/lib/kubelet/pods/0d2366ff-2ee5-4ef5-b9f8-5605a1337924/volumes" Nov 02 18:00:21.510821 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.510794 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54f4f507-8d94-412d-9189-7ac2a4531f01" path="/var/lib/kubelet/pods/54f4f507-8d94-412d-9189-7ac2a4531f01/volumes" Nov 02 18:00:21.513344 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.513315 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h"] Nov 02 18:00:21.597631 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.597600 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv"] Nov 02 18:00:21.604236 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.604210 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6"] Nov 02 18:00:21.606349 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.606325 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" Nov 02 18:00:21.611020 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.610991 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894040-sv9x6"] Nov 02 18:00:21.760986 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.760903 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1cc711d-aed9-4bdc-84b8-795dfc52a7bd-kube-api-access-w7zbx\" (UniqueName: \"kubernetes.io/projected/f1cc711d-aed9-4bdc-84b8-795dfc52a7bd-kube-api-access-w7zbx\") pod \"f1cc711d-aed9-4bdc-84b8-795dfc52a7bd\" (UID: \"f1cc711d-aed9-4bdc-84b8-795dfc52a7bd\") " Nov 02 18:00:21.761519 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.761491 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1cc711d-aed9-4bdc-84b8-795dfc52a7bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1cc711d-aed9-4bdc-84b8-795dfc52a7bd-maas-api-service-ca\") pod \"f1cc711d-aed9-4bdc-84b8-795dfc52a7bd\" (UID: \"f1cc711d-aed9-4bdc-84b8-795dfc52a7bd\") " Nov 02 18:00:21.761893 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.761841 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1cc711d-aed9-4bdc-84b8-795dfc52a7bd-maas-api-service-ca" pod "f1cc711d-aed9-4bdc-84b8-795dfc52a7bd" (UID: "f1cc711d-aed9-4bdc-84b8-795dfc52a7bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:21.762519 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.762498 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1cc711d-aed9-4bdc-84b8-795dfc52a7bd-maas-api-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:21.763302 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.763266 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1cc711d-aed9-4bdc-84b8-795dfc52a7bd-kube-api-access-w7zbx" pod "f1cc711d-aed9-4bdc-84b8-795dfc52a7bd" (UID: "f1cc711d-aed9-4bdc-84b8-795dfc52a7bd"). InnerVolumeSpecName "kube-api-access-w7zbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:21.777298 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:21.777072 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:00:21.864180 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.864149 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7zbx\" (UniqueName: \"kubernetes.io/projected/f1cc711d-aed9-4bdc-84b8-795dfc52a7bd-kube-api-access-w7zbx\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:21.920518 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:21.920485 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:22.365591 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.365565 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" Nov 02 18:00:22.365726 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.365605 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv" event={"ID":"f1cc711d-aed9-4bdc-84b8-795dfc52a7bd","Type":"ContainerDied","Data":"e43cab02a24b61b21bbf35f2839756f5e2c7448a4da3ff19ea43476579586b8e"} Nov 02 18:00:22.365726 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.365668 4860 scope.go:122] "RemoveContainer" containerID="1f21a8826a17bfe12298a35acdc08a92ed4531f56f1e2f111db573722f4971bd" Nov 02 18:00:22.494135 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.493255 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv"] Nov 02 18:00:22.508381 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.508341 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d5b905-29894040-sx2gv"] Nov 02 18:00:22.575609 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.575572 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" Nov 02 18:00:22.633988 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.633916 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" Nov 02 18:00:22.678682 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.677368 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/487dac52-8fbc-425e-af6c-3db7403709ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/487dac52-8fbc-425e-af6c-3db7403709ff-maas-api-service-ca\") pod \"487dac52-8fbc-425e-af6c-3db7403709ff\" (UID: \"487dac52-8fbc-425e-af6c-3db7403709ff\") " Nov 02 18:00:22.678682 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.677645 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/487dac52-8fbc-425e-af6c-3db7403709ff-kube-api-access-vb6ww\" (UniqueName: \"kubernetes.io/projected/487dac52-8fbc-425e-af6c-3db7403709ff-kube-api-access-vb6ww\") pod \"487dac52-8fbc-425e-af6c-3db7403709ff\" (UID: \"487dac52-8fbc-425e-af6c-3db7403709ff\") " Nov 02 18:00:22.678682 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.677682 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b76941-3104-4e66-a6b0-52a20c9eb4f5-kube-api-access-79vh5\" (UniqueName: \"kubernetes.io/projected/65b76941-3104-4e66-a6b0-52a20c9eb4f5-kube-api-access-79vh5\") pod \"65b76941-3104-4e66-a6b0-52a20c9eb4f5\" (UID: \"65b76941-3104-4e66-a6b0-52a20c9eb4f5\") " Nov 02 18:00:22.678682 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.677740 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65b76941-3104-4e66-a6b0-52a20c9eb4f5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b76941-3104-4e66-a6b0-52a20c9eb4f5-maas-api-service-ca\") pod \"65b76941-3104-4e66-a6b0-52a20c9eb4f5\" (UID: \"65b76941-3104-4e66-a6b0-52a20c9eb4f5\") " Nov 02 18:00:22.678682 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.677757 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/487dac52-8fbc-425e-af6c-3db7403709ff-maas-api-service-ca" pod "487dac52-8fbc-425e-af6c-3db7403709ff" (UID: "487dac52-8fbc-425e-af6c-3db7403709ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:22.678682 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.678456 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b76941-3104-4e66-a6b0-52a20c9eb4f5-maas-api-service-ca" pod "65b76941-3104-4e66-a6b0-52a20c9eb4f5" (UID: "65b76941-3104-4e66-a6b0-52a20c9eb4f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:22.678682 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.678630 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/487dac52-8fbc-425e-af6c-3db7403709ff-maas-api-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:22.678682 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.678653 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b76941-3104-4e66-a6b0-52a20c9eb4f5-maas-api-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:22.680370 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.680347 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487dac52-8fbc-425e-af6c-3db7403709ff-kube-api-access-vb6ww" pod "487dac52-8fbc-425e-af6c-3db7403709ff" (UID: "487dac52-8fbc-425e-af6c-3db7403709ff"). InnerVolumeSpecName "kube-api-access-vb6ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:22.680494 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.680441 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b76941-3104-4e66-a6b0-52a20c9eb4f5-kube-api-access-79vh5" pod "65b76941-3104-4e66-a6b0-52a20c9eb4f5" (UID: "65b76941-3104-4e66-a6b0-52a20c9eb4f5"). InnerVolumeSpecName "kube-api-access-79vh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:22.780201 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.780169 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb6ww\" (UniqueName: \"kubernetes.io/projected/487dac52-8fbc-425e-af6c-3db7403709ff-kube-api-access-vb6ww\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:22.780201 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.780198 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79vh5\" (UniqueName: \"kubernetes.io/projected/65b76941-3104-4e66-a6b0-52a20c9eb4f5-kube-api-access-79vh5\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:22.923803 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.923673 4860 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 18:00:22.923803 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:22.923746 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" podUID="5b585147-38f8-4a8f-be0b-d1ffaef9481d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 18:00:23.379367 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:23.379336 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" Nov 02 18:00:23.379542 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:23.379387 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h" event={"ID":"487dac52-8fbc-425e-af6c-3db7403709ff","Type":"ContainerDied","Data":"ba3b91c304e57217e5b3aa06366ab70b5b0b7ab64154944768869ca152633235"} Nov 02 18:00:23.379542 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:23.379492 4860 scope.go:122] "RemoveContainer" containerID="624de86b97fca222861cf1ae09c22072bed1b6acc1edef899866b704a81a535f" Nov 02 18:00:23.382517 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:23.382491 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" Nov 02 18:00:23.382661 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:23.382532 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv" event={"ID":"65b76941-3104-4e66-a6b0-52a20c9eb4f5","Type":"ContainerDied","Data":"3feb563fbbf0a82ace7f8aa7820f11dc106a218553c6a92c5a47f89caecd8cb6"} Nov 02 18:00:23.392140 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:23.392109 4860 scope.go:122] "RemoveContainer" containerID="f620515b015815da21213c282dd8605f0d9e511d4c97cc3bc27fdfd39375cb04" Nov 02 18:00:23.434484 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:23.434451 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h"] Nov 02 18:00:23.440104 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:23.440078 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894040-d2s2h"] Nov 02 18:00:23.472011 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:23.471980 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv"] Nov 02 18:00:23.481094 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:23.481068 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2672ed-29894040-gmqdv"] Nov 02 18:00:23.504173 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:23.504140 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ffcc622-4641-4348-ad7a-88797d8b2469" path="/var/lib/kubelet/pods/0ffcc622-4641-4348-ad7a-88797d8b2469/volumes" Nov 02 18:00:23.505026 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:23.505003 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="487dac52-8fbc-425e-af6c-3db7403709ff" path="/var/lib/kubelet/pods/487dac52-8fbc-425e-af6c-3db7403709ff/volumes" Nov 02 18:00:23.505793 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:23.505773 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65b76941-3104-4e66-a6b0-52a20c9eb4f5" path="/var/lib/kubelet/pods/65b76941-3104-4e66-a6b0-52a20c9eb4f5/volumes" Nov 02 18:00:23.507001 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:23.506981 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1cc711d-aed9-4bdc-84b8-795dfc52a7bd" path="/var/lib/kubelet/pods/f1cc711d-aed9-4bdc-84b8-795dfc52a7bd/volumes" Nov 02 18:00:23.920324 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:23.920289 4860 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 18:00:23.920782 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:23.920343 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" podUID="5b585147-38f8-4a8f-be0b-d1ffaef9481d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 18:00:24.920448 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:24.920416 4860 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 18:00:24.920865 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:24.920469 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" podUID="5b585147-38f8-4a8f-be0b-d1ffaef9481d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 18:00:25.323061 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:25.323023 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" Nov 02 18:00:25.920405 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:25.920373 4860 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 18:00:25.920568 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:25.920427 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" podUID="5b585147-38f8-4a8f-be0b-d1ffaef9481d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 18:00:26.921072 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:26.921036 4860 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 18:00:26.921500 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:26.921101 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" podUID="5b585147-38f8-4a8f-be0b-d1ffaef9481d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 18:00:27.920478 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:27.920443 4860 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 18:00:27.920629 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:27.920496 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" podUID="5b585147-38f8-4a8f-be0b-d1ffaef9481d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 18:00:28.921041 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:28.921002 4860 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 18:00:28.921425 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:28.921070 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" podUID="5b585147-38f8-4a8f-be0b-d1ffaef9481d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 18:00:29.920826 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:29.920787 4860 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 18:00:29.921085 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:29.920843 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" podUID="5b585147-38f8-4a8f-be0b-d1ffaef9481d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 18:00:30.925906 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:30.925865 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:30.926393 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:30.926372 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:30.927096 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:30.927071 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:00:33.233270 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.233233 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m"] Nov 02 18:00:33.234297 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234272 4860 cpu_manager.go:414] "containerMap: removing container" podUID="54f4f507-8d94-412d-9189-7ac2a4531f01" containerName="cleanup" Nov 02 18:00:33.234297 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234297 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f4f507-8d94-412d-9189-7ac2a4531f01" containerName="cleanup" Nov 02 18:00:33.234483 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234327 4860 cpu_manager.go:414] "containerMap: removing container" podUID="f1cc711d-aed9-4bdc-84b8-795dfc52a7bd" containerName="cleanup" Nov 02 18:00:33.234483 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234335 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1cc711d-aed9-4bdc-84b8-795dfc52a7bd" containerName="cleanup" Nov 02 18:00:33.234483 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234344 4860 cpu_manager.go:414] "containerMap: removing container" podUID="65b76941-3104-4e66-a6b0-52a20c9eb4f5" containerName="cleanup" Nov 02 18:00:33.234483 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234351 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b76941-3104-4e66-a6b0-52a20c9eb4f5" containerName="cleanup" Nov 02 18:00:33.234483 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234369 4860 cpu_manager.go:414] "containerMap: removing container" podUID="f1cc711d-aed9-4bdc-84b8-795dfc52a7bd" containerName="cleanup" Nov 02 18:00:33.234483 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234376 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1cc711d-aed9-4bdc-84b8-795dfc52a7bd" containerName="cleanup" Nov 02 18:00:33.234483 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234395 4860 cpu_manager.go:414] "containerMap: removing container" podUID="65b76941-3104-4e66-a6b0-52a20c9eb4f5" containerName="cleanup" Nov 02 18:00:33.234483 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234402 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b76941-3104-4e66-a6b0-52a20c9eb4f5" containerName="cleanup" Nov 02 18:00:33.234483 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234420 4860 cpu_manager.go:414] "containerMap: removing container" podUID="487dac52-8fbc-425e-af6c-3db7403709ff" containerName="cleanup" Nov 02 18:00:33.234483 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234426 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="487dac52-8fbc-425e-af6c-3db7403709ff" containerName="cleanup" Nov 02 18:00:33.234483 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234447 4860 cpu_manager.go:414] "containerMap: removing container" podUID="65b76941-3104-4e66-a6b0-52a20c9eb4f5" containerName="cleanup" Nov 02 18:00:33.234483 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234454 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b76941-3104-4e66-a6b0-52a20c9eb4f5" containerName="cleanup" Nov 02 18:00:33.234483 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234474 4860 cpu_manager.go:414] "containerMap: removing container" podUID="0d2366ff-2ee5-4ef5-b9f8-5605a1337924" containerName="cleanup" Nov 02 18:00:33.234483 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234481 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2366ff-2ee5-4ef5-b9f8-5605a1337924" containerName="cleanup" Nov 02 18:00:33.235212 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234529 4860 cpu_manager.go:414] "containerMap: removing container" podUID="487dac52-8fbc-425e-af6c-3db7403709ff" containerName="cleanup" Nov 02 18:00:33.235212 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234537 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="487dac52-8fbc-425e-af6c-3db7403709ff" containerName="cleanup" Nov 02 18:00:33.235212 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234555 4860 cpu_manager.go:414] "containerMap: removing container" podUID="487dac52-8fbc-425e-af6c-3db7403709ff" containerName="cleanup" Nov 02 18:00:33.235212 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234563 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="487dac52-8fbc-425e-af6c-3db7403709ff" containerName="cleanup" Nov 02 18:00:33.235212 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234759 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1cc711d-aed9-4bdc-84b8-795dfc52a7bd" containerName="cleanup" Nov 02 18:00:33.235212 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234775 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2366ff-2ee5-4ef5-b9f8-5605a1337924" containerName="cleanup" Nov 02 18:00:33.235212 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234792 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ffcc622-4641-4348-ad7a-88797d8b2469" containerName="cleanup" Nov 02 18:00:33.235212 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234803 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="487dac52-8fbc-425e-af6c-3db7403709ff" containerName="cleanup" Nov 02 18:00:33.235212 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234812 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f4f507-8d94-412d-9189-7ac2a4531f01" containerName="cleanup" Nov 02 18:00:33.235212 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234825 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1cc711d-aed9-4bdc-84b8-795dfc52a7bd" containerName="cleanup" Nov 02 18:00:33.235212 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234832 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b76941-3104-4e66-a6b0-52a20c9eb4f5" containerName="cleanup" Nov 02 18:00:33.235212 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234860 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1cc711d-aed9-4bdc-84b8-795dfc52a7bd" containerName="cleanup" Nov 02 18:00:33.235212 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234870 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b76941-3104-4e66-a6b0-52a20c9eb4f5" containerName="cleanup" Nov 02 18:00:33.235212 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234882 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="487dac52-8fbc-425e-af6c-3db7403709ff" containerName="cleanup" Nov 02 18:00:33.235212 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.234894 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b76941-3104-4e66-a6b0-52a20c9eb4f5" containerName="cleanup" Nov 02 18:00:33.240751 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.240728 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.263368 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.263341 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m"] Nov 02 18:00:33.399410 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.399363 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef174d1-f0e7-4748-997a-fa814937890c-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.399785 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.399743 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.400105 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.399910 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.400181 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.400095 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef174d1-f0e7-4748-997a-fa814937890c-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.400181 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.400147 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.400300 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.400199 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.400300 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.400267 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef174d1-f0e7-4748-997a-fa814937890c-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.400474 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.400306 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.400474 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.400440 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvndd\" (UniqueName: \"kubernetes.io/projected/cef174d1-f0e7-4748-997a-fa814937890c-kube-api-access-rvndd\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.502019 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.501934 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.502179 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.502029 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.502179 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.502110 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef174d1-f0e7-4748-997a-fa814937890c-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.502179 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.502143 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.502373 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.502181 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.502373 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.502243 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef174d1-f0e7-4748-997a-fa814937890c-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.502373 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.502280 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.502373 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.502369 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvndd\" (UniqueName: \"kubernetes.io/projected/cef174d1-f0e7-4748-997a-fa814937890c-kube-api-access-rvndd\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.502582 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.502497 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef174d1-f0e7-4748-997a-fa814937890c-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.503314 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.502962 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.503314 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.503011 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.503314 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.503153 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.503631 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.503605 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef174d1-f0e7-4748-997a-fa814937890c-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.503741 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.503715 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.505007 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.504985 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.505778 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.505754 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef174d1-f0e7-4748-997a-fa814937890c-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.525620 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.525590 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvndd\" (UniqueName: \"kubernetes.io/projected/cef174d1-f0e7-4748-997a-fa814937890c-kube-api-access-rvndd\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.525620 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.525609 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef174d1-f0e7-4748-997a-fa814937890c-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.556134 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.556105 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:33.915391 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:00:33.915352 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef174d1_f0e7_4748_997a_fa814937890c.slice/crio-9f4fb8609053c2db065f0d53c6302af7ebe67229a0e89135d34115f421a7c11f WatchSource:0}: Error finding container 9f4fb8609053c2db065f0d53c6302af7ebe67229a0e89135d34115f421a7c11f: Status 404 returned error can't find the container with id 9f4fb8609053c2db065f0d53c6302af7ebe67229a0e89135d34115f421a7c11f Nov 02 18:00:33.916439 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.916415 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m"] Nov 02 18:00:33.918560 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.918523 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:00:33.918666 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.918605 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:00:33.918666 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:33.918651 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:00:34.555555 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:34.555513 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" event={"ID":"cef174d1-f0e7-4748-997a-fa814937890c","Type":"ContainerStarted","Data":"04b2682221e9d31acfba1c6db2f8298d1ebbd56c142cbd668430517d23a4d90b"} Nov 02 18:00:34.555961 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:34.555612 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" event={"ID":"cef174d1-f0e7-4748-997a-fa814937890c","Type":"ContainerStarted","Data":"9f4fb8609053c2db065f0d53c6302af7ebe67229a0e89135d34115f421a7c11f"} Nov 02 18:00:34.556776 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:34.556755 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:34.613452 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:34.613392 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" podStartSLOduration=1.613375234 podStartE2EDuration="1.613375234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:34.6122872 +0000 UTC m=+1853.233646975" watchObservedRunningTime="2026-11-02 18:00:34.613375234 +0000 UTC m=+1853.234735017" Nov 02 18:00:35.579134 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:35.579085 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 02 18:00:35.579703 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:35.579169 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" podUID="cef174d1-f0e7-4748-997a-fa814937890c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 02 18:00:36.574874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.566345 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 02 18:00:36.579528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.578976 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" podUID="cef174d1-f0e7-4748-997a-fa814937890c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 02 18:00:36.589800 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.589768 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg"] Nov 02 18:00:36.590660 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.590642 4860 cpu_manager.go:414] "containerMap: removing container" podUID="f1cc711d-aed9-4bdc-84b8-795dfc52a7bd" containerName="cleanup" Nov 02 18:00:36.590660 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.590661 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1cc711d-aed9-4bdc-84b8-795dfc52a7bd" containerName="cleanup" Nov 02 18:00:36.590831 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.590821 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="487dac52-8fbc-425e-af6c-3db7403709ff" containerName="cleanup" Nov 02 18:00:36.595411 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.595389 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" Nov 02 18:00:36.612838 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.612810 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-fbbbf8-serving-cert\"" Nov 02 18:00:36.618055 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.618023 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg"] Nov 02 18:00:36.792887 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.792814 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpl5r\" (UniqueName: \"kubernetes.io/projected/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-kube-api-access-vpl5r\") pod \"maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg\" (UID: \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" Nov 02 18:00:36.793058 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.792949 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg\" (UID: \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" Nov 02 18:00:36.793130 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.793106 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-maas-api-tls\") pod \"maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg\" (UID: \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" Nov 02 18:00:36.793427 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.793411 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-metrics-certs\") pod \"maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg\" (UID: \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" Nov 02 18:00:36.900949 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.896595 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-metrics-certs\") pod \"maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg\" (UID: \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" Nov 02 18:00:36.903883 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.901172 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpl5r\" (UniqueName: \"kubernetes.io/projected/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-kube-api-access-vpl5r\") pod \"maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg\" (UID: \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" Nov 02 18:00:36.903883 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.901265 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg\" (UID: \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" Nov 02 18:00:36.909213 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.904277 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-maas-api-tls\") pod \"maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg\" (UID: \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" Nov 02 18:00:36.911672 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.910656 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg\" (UID: \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" Nov 02 18:00:36.916921 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.916893 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-metrics-certs\") pod \"maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg\" (UID: \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" Nov 02 18:00:36.919817 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.919787 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-maas-api-tls\") pod \"maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg\" (UID: \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" Nov 02 18:00:36.950934 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:36.948618 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpl5r\" (UniqueName: \"kubernetes.io/projected/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-kube-api-access-vpl5r\") pod \"maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg\" (UID: \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" Nov 02 18:00:37.215750 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:37.215663 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" Nov 02 18:00:37.558876 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:37.558228 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 02 18:00:37.558876 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:37.558291 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" podUID="cef174d1-f0e7-4748-997a-fa814937890c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 02 18:00:37.621405 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:37.621374 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg"] Nov 02 18:00:38.579918 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:38.579045 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 02 18:00:38.579918 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:38.579129 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" podUID="cef174d1-f0e7-4748-997a-fa814937890c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 02 18:00:38.624493 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:38.624448 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" event={"ID":"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1","Type":"ContainerStarted","Data":"237a5fa79b8e8ff9f658274b472e3363be975f4d890f0a9f0cdce21fad61b343"} Nov 02 18:00:38.625039 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:38.624570 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" event={"ID":"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1","Type":"ContainerStarted","Data":"16fc2400c0eed3c4d77a869b1785ba808df0f6f670d0975f3b318d969afaa689"} Nov 02 18:00:38.625039 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:38.624656 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" Nov 02 18:00:38.690244 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:38.690181 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" podStartSLOduration=2.170574617 podStartE2EDuration="2.690164405s" totalImagesPullingTime="519.589788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:38.688042305 +0000 UTC m=+1857.309402075" watchObservedRunningTime="2026-11-02 18:00:38.690164405 +0000 UTC m=+1857.311524172" Nov 02 18:00:39.579507 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:39.579462 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 02 18:00:39.579721 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:39.579544 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" podUID="cef174d1-f0e7-4748-997a-fa814937890c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 02 18:00:40.557447 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:40.557413 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 02 18:00:40.557869 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:40.557466 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" podUID="cef174d1-f0e7-4748-997a-fa814937890c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 02 18:00:41.563342 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:41.563304 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:41.563779 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:41.563764 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:41.564578 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:41.564561 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:00:42.952916 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:42.952830 4860 scope.go:122] "RemoveContainer" containerID="cdbb516d9028519d96d19562e7bae8d5ea0b92ec7fd8f8ea9858bd4dbd2177ed" Nov 02 18:00:44.700872 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:44.693089 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" Nov 02 18:00:49.774585 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:49.774551 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4"] Nov 02 18:00:49.775094 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:49.774750 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" podUID="da05c910-1ef7-47e6-a5bf-ead3b63064d8" containerName="istio-proxy" containerID="cri-o://ce2df665bf6d53a0003f3558f90086f5fc27f9b115b78810cadf5fb7d8533973" gracePeriod=30 Nov 02 18:00:50.583907 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.583869 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8"] Nov 02 18:00:50.584137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.584096 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" podUID="0ebc4571-d5c2-45f7-80de-5f79648303f4" containerName="istio-proxy" containerID="cri-o://6b6c82d53d03ce3c28d7c2f39df0f2fdb88747cce90319002e49e4b86c77f9b5" gracePeriod=30 Nov 02 18:00:50.708415 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.708374 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr"] Nov 02 18:00:50.715166 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.715141 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.747651 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.747619 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr"] Nov 02 18:00:50.838954 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.838835 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-credential-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.839357 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.838962 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istiod-ca-cert\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.839357 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.839118 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-envoy\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.839357 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.839174 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-workload-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.839357 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.839321 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zccp\" (UniqueName: \"kubernetes.io/projected/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-kube-api-access-4zccp\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.839357 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.839348 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-token\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.839552 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.839368 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-podinfo\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.839552 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.839393 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-data\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.839552 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.839547 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-workload-certs\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.941291 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.941257 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-envoy\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.941458 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.941317 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-workload-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.941458 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.941426 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zccp\" (UniqueName: \"kubernetes.io/projected/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-kube-api-access-4zccp\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.941458 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.941449 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-token\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.941589 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.941468 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-podinfo\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.941589 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.941491 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-data\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.941692 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.941644 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-workload-certs\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.941751 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.941711 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-credential-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.941818 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.941750 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istiod-ca-cert\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.941818 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.941789 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-workload-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.942044 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.942008 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-data\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.942110 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.942080 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-workload-certs\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.942196 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.942171 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-credential-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.942463 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.942445 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istiod-ca-cert\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.943731 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.943706 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-envoy\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.943995 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.943976 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-podinfo\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.965387 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.965352 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-token\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:50.965564 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:50.965537 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zccp\" (UniqueName: \"kubernetes.io/projected/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-kube-api-access-4zccp\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-jdhvr\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:51.049912 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:51.049878 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:51.379991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:51.379957 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr"] Nov 02 18:00:51.382324 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:51.382286 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:00:51.382441 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:51.382370 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:00:51.382517 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:51.382446 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:00:51.780817 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:00:51.780717 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:00:51.879520 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:51.879468 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" event={"ID":"76282517-6c00-4e83-b5f5-a6cfaa0dcad2","Type":"ContainerStarted","Data":"71da5e1a61cd83b0c52aaa1465cd9942dbe20a8750bf5364a1e8f96863c566ae"} Nov 02 18:00:51.880049 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:51.879551 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" event={"ID":"76282517-6c00-4e83-b5f5-a6cfaa0dcad2","Type":"ContainerStarted","Data":"7622afd83d6bcd6424e3e40d18ef54c095dbdd326d1cb6bb0b338cb614fabf11"} Nov 02 18:00:51.929026 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:51.928971 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" podStartSLOduration=1.928956967 podStartE2EDuration="1.928956967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:51.925109859 +0000 UTC m=+1870.546469635" watchObservedRunningTime="2026-11-02 18:00:51.928956967 +0000 UTC m=+1870.550316740" Nov 02 18:00:52.050758 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:52.050664 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:52.052319 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:52.052291 4860 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 02 18:00:52.052439 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:52.052351 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" podUID="76282517-6c00-4e83-b5f5-a6cfaa0dcad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 02 18:00:53.050886 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:53.050837 4860 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 02 18:00:53.051385 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:53.050910 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" podUID="76282517-6c00-4e83-b5f5-a6cfaa0dcad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 02 18:00:53.747535 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:53.747496 4860 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:00:53.747719 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:53.747552 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" podUID="da05c910-1ef7-47e6-a5bf-ead3b63064d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:00:54.050498 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:54.050414 4860 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 02 18:00:54.050498 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:54.050469 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" podUID="76282517-6c00-4e83-b5f5-a6cfaa0dcad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 02 18:00:54.915798 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:54.915760 4860 generic.go:356] "Generic (PLEG): container finished" podID="da05c910-1ef7-47e6-a5bf-ead3b63064d8" containerID="ce2df665bf6d53a0003f3558f90086f5fc27f9b115b78810cadf5fb7d8533973" exitCode=0 Nov 02 18:00:54.916309 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:54.915824 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" event={"ID":"da05c910-1ef7-47e6-a5bf-ead3b63064d8","Type":"ContainerDied","Data":"ce2df665bf6d53a0003f3558f90086f5fc27f9b115b78810cadf5fb7d8533973"} Nov 02 18:00:55.059934 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.052171 4860 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 02 18:00:55.059934 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.052220 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" podUID="76282517-6c00-4e83-b5f5-a6cfaa0dcad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 02 18:00:55.152244 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.152216 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 18:00:55.249737 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.249697 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-podinfo\") pod \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " Nov 02 18:00:55.249737 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.249752 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istiod-ca-cert\") pod \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " Nov 02 18:00:55.250019 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.249786 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-envoy\") pod \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " Nov 02 18:00:55.250019 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.249914 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-token\" (UniqueName: \"kubernetes.io/projected/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-token\") pod \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " Nov 02 18:00:55.250149 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.250023 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da05c910-1ef7-47e6-a5bf-ead3b63064d8-kube-api-access-g5jf4\" (UniqueName: \"kubernetes.io/projected/da05c910-1ef7-47e6-a5bf-ead3b63064d8-kube-api-access-g5jf4\") pod \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " Nov 02 18:00:55.250149 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.250136 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-data\") pod \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " Nov 02 18:00:55.250149 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.250132 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istiod-ca-cert" pod "da05c910-1ef7-47e6-a5bf-ead3b63064d8" (UID: "da05c910-1ef7-47e6-a5bf-ead3b63064d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:55.250308 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.250187 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-workload-certs\") pod \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " Nov 02 18:00:55.250308 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.250232 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-credential-socket\") pod \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " Nov 02 18:00:55.250308 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.250304 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-workload-socket\") pod \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\" (UID: \"da05c910-1ef7-47e6-a5bf-ead3b63064d8\") " Nov 02 18:00:55.250536 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.250504 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-data" pod "da05c910-1ef7-47e6-a5bf-ead3b63064d8" (UID: "da05c910-1ef7-47e6-a5bf-ead3b63064d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:55.250592 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.250526 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-workload-certs" pod "da05c910-1ef7-47e6-a5bf-ead3b63064d8" (UID: "da05c910-1ef7-47e6-a5bf-ead3b63064d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:55.250758 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.250734 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-workload-socket" pod "da05c910-1ef7-47e6-a5bf-ead3b63064d8" (UID: "da05c910-1ef7-47e6-a5bf-ead3b63064d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:55.251084 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.251058 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-credential-socket" pod "da05c910-1ef7-47e6-a5bf-ead3b63064d8" (UID: "da05c910-1ef7-47e6-a5bf-ead3b63064d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:55.251297 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.251273 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:55.251384 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.251304 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:55.251384 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.251319 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:55.251384 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.251335 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:55.251384 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.251349 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:55.253373 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.253350 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da05c910-1ef7-47e6-a5bf-ead3b63064d8-kube-api-access-g5jf4" pod "da05c910-1ef7-47e6-a5bf-ead3b63064d8" (UID: "da05c910-1ef7-47e6-a5bf-ead3b63064d8"). InnerVolumeSpecName "kube-api-access-g5jf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:55.253482 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.253378 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-token" pod "da05c910-1ef7-47e6-a5bf-ead3b63064d8" (UID: "da05c910-1ef7-47e6-a5bf-ead3b63064d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:55.253596 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.253581 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-envoy" pod "da05c910-1ef7-47e6-a5bf-ead3b63064d8" (UID: "da05c910-1ef7-47e6-a5bf-ead3b63064d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:55.286526 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.286477 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-podinfo" pod "da05c910-1ef7-47e6-a5bf-ead3b63064d8" (UID: "da05c910-1ef7-47e6-a5bf-ead3b63064d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:00:55.353158 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.353083 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:55.353158 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.353112 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:55.353158 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.353121 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da05c910-1ef7-47e6-a5bf-ead3b63064d8-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:55.353158 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.353129 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5jf4\" (UniqueName: \"kubernetes.io/projected/da05c910-1ef7-47e6-a5bf-ead3b63064d8-kube-api-access-g5jf4\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:55.933390 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.933358 4860 generic.go:356] "Generic (PLEG): container finished" podID="0ebc4571-d5c2-45f7-80de-5f79648303f4" containerID="6b6c82d53d03ce3c28d7c2f39df0f2fdb88747cce90319002e49e4b86c77f9b5" exitCode=0 Nov 02 18:00:55.933814 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.933441 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" event={"ID":"0ebc4571-d5c2-45f7-80de-5f79648303f4","Type":"ContainerDied","Data":"6b6c82d53d03ce3c28d7c2f39df0f2fdb88747cce90319002e49e4b86c77f9b5"} Nov 02 18:00:55.933814 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.933510 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" event={"ID":"0ebc4571-d5c2-45f7-80de-5f79648303f4","Type":"ContainerDied","Data":"c5cf8b42508774ab4cd6f57e004b8f6b2d18cf50b75e8c6b73d7c2fc29810ca4"} Nov 02 18:00:55.933814 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.933534 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5cf8b42508774ab4cd6f57e004b8f6b2d18cf50b75e8c6b73d7c2fc29810ca4" Nov 02 18:00:55.935327 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.935292 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" event={"ID":"da05c910-1ef7-47e6-a5bf-ead3b63064d8","Type":"ContainerDied","Data":"b8467dc11ce269c295dcde035ff6f9d8041afe580f8b6573ad05adb7b68269a1"} Nov 02 18:00:55.935423 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.935360 4860 scope.go:122] "RemoveContainer" containerID="ce2df665bf6d53a0003f3558f90086f5fc27f9b115b78810cadf5fb7d8533973" Nov 02 18:00:55.935531 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.935519 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4" Nov 02 18:00:55.949479 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:55.949456 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 18:00:56.021926 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.021830 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4"] Nov 02 18:00:56.041632 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.041596 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-hkrk4"] Nov 02 18:00:56.050704 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.050674 4860 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 02 18:00:56.050871 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.050745 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" podUID="76282517-6c00-4e83-b5f5-a6cfaa0dcad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 02 18:00:56.096812 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.096780 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-credential-socket\") pod \"0ebc4571-d5c2-45f7-80de-5f79648303f4\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " Nov 02 18:00:56.097019 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.096828 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-workload-certs\") pod \"0ebc4571-d5c2-45f7-80de-5f79648303f4\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " Nov 02 18:00:56.097019 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.096871 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-token\" (UniqueName: \"kubernetes.io/projected/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-token\") pod \"0ebc4571-d5c2-45f7-80de-5f79648303f4\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " Nov 02 18:00:56.097019 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.096950 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-workload-socket\") pod \"0ebc4571-d5c2-45f7-80de-5f79648303f4\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " Nov 02 18:00:56.097019 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.096970 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-podinfo\") pod \"0ebc4571-d5c2-45f7-80de-5f79648303f4\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " Nov 02 18:00:56.097250 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.097061 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ebc4571-d5c2-45f7-80de-5f79648303f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebc4571-d5c2-45f7-80de-5f79648303f4-istiod-ca-cert\") pod \"0ebc4571-d5c2-45f7-80de-5f79648303f4\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " Nov 02 18:00:56.097250 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.097105 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-credential-socket" pod "0ebc4571-d5c2-45f7-80de-5f79648303f4" (UID: "0ebc4571-d5c2-45f7-80de-5f79648303f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:56.097250 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.097156 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebc4571-d5c2-45f7-80de-5f79648303f4-kube-api-access-p542m\" (UniqueName: \"kubernetes.io/projected/0ebc4571-d5c2-45f7-80de-5f79648303f4-kube-api-access-p542m\") pod \"0ebc4571-d5c2-45f7-80de-5f79648303f4\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " Nov 02 18:00:56.097250 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.097175 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-data\") pod \"0ebc4571-d5c2-45f7-80de-5f79648303f4\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " Nov 02 18:00:56.097250 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.097227 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-envoy\") pod \"0ebc4571-d5c2-45f7-80de-5f79648303f4\" (UID: \"0ebc4571-d5c2-45f7-80de-5f79648303f4\") " Nov 02 18:00:56.098038 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.098012 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:56.098222 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.098200 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-workload-certs" pod "0ebc4571-d5c2-45f7-80de-5f79648303f4" (UID: "0ebc4571-d5c2-45f7-80de-5f79648303f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:56.098306 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.098274 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-data" pod "0ebc4571-d5c2-45f7-80de-5f79648303f4" (UID: "0ebc4571-d5c2-45f7-80de-5f79648303f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:56.098500 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.098483 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-workload-socket" pod "0ebc4571-d5c2-45f7-80de-5f79648303f4" (UID: "0ebc4571-d5c2-45f7-80de-5f79648303f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:56.098739 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.098721 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ebc4571-d5c2-45f7-80de-5f79648303f4-istiod-ca-cert" pod "0ebc4571-d5c2-45f7-80de-5f79648303f4" (UID: "0ebc4571-d5c2-45f7-80de-5f79648303f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:56.099596 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.099573 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-token" pod "0ebc4571-d5c2-45f7-80de-5f79648303f4" (UID: "0ebc4571-d5c2-45f7-80de-5f79648303f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:56.101441 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.100770 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-envoy" pod "0ebc4571-d5c2-45f7-80de-5f79648303f4" (UID: "0ebc4571-d5c2-45f7-80de-5f79648303f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:56.101441 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.100889 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebc4571-d5c2-45f7-80de-5f79648303f4-kube-api-access-p542m" pod "0ebc4571-d5c2-45f7-80de-5f79648303f4" (UID: "0ebc4571-d5c2-45f7-80de-5f79648303f4"). InnerVolumeSpecName "kube-api-access-p542m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:56.103577 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.102975 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-podinfo" pod "0ebc4571-d5c2-45f7-80de-5f79648303f4" (UID: "0ebc4571-d5c2-45f7-80de-5f79648303f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:00:56.200668 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.200629 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p542m\" (UniqueName: \"kubernetes.io/projected/0ebc4571-d5c2-45f7-80de-5f79648303f4-kube-api-access-p542m\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:56.200668 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.200668 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:56.200894 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.200683 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:56.200894 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.200697 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:56.200894 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.200709 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:56.200894 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.200721 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc4571-d5c2-45f7-80de-5f79648303f4-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:56.200894 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.200734 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebc4571-d5c2-45f7-80de-5f79648303f4-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:56.200894 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.200748 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebc4571-d5c2-45f7-80de-5f79648303f4-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:00:56.944916 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:56.944885 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8" Nov 02 18:00:57.073869 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:57.054168 4860 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 02 18:00:57.073869 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:57.054244 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" podUID="76282517-6c00-4e83-b5f5-a6cfaa0dcad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 02 18:00:57.100885 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:57.099929 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8"] Nov 02 18:00:57.117240 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:57.115836 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-p58l8"] Nov 02 18:00:57.503251 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:57.503213 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ebc4571-d5c2-45f7-80de-5f79648303f4" path="/var/lib/kubelet/pods/0ebc4571-d5c2-45f7-80de-5f79648303f4/volumes" Nov 02 18:00:57.503941 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:57.503917 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da05c910-1ef7-47e6-a5bf-ead3b63064d8" path="/var/lib/kubelet/pods/da05c910-1ef7-47e6-a5bf-ead3b63064d8/volumes" Nov 02 18:00:58.055930 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:58.055896 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:58.056393 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:58.056360 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:00:58.057178 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:00:58.057155 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:01:06.451202 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:06.451170 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf"] Nov 02 18:01:21.493638 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:21.493402 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" podUID="0281a566-f641-4a84-aa07-03f6e6845bc1" containerName="main" containerID="cri-o://93e924edf3f941dbf758f5ce5e7ffab8586236d1ed6018e8ab1dfeb23fa305f8" gracePeriod=45 Nov 02 18:01:21.939408 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:01:21.939268 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:01:21.978345 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:21.978302 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" podUID="0281a566-f641-4a84-aa07-03f6e6845bc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.159:8000/ready\": dial tcp 10.132.0.159:8000: connect: connection refused" Nov 02 18:01:22.291088 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.291049 4860 generic.go:356] "Generic (PLEG): container finished" podID="0281a566-f641-4a84-aa07-03f6e6845bc1" containerID="93e924edf3f941dbf758f5ce5e7ffab8586236d1ed6018e8ab1dfeb23fa305f8" exitCode=0 Nov 02 18:01:22.291255 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.291121 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" event={"ID":"0281a566-f641-4a84-aa07-03f6e6845bc1","Type":"ContainerDied","Data":"93e924edf3f941dbf758f5ce5e7ffab8586236d1ed6018e8ab1dfeb23fa305f8"} Nov 02 18:01:22.456842 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.456779 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 18:01:22.483392 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.483356 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0281a566-f641-4a84-aa07-03f6e6845bc1-tls-certs\" (UniqueName: \"kubernetes.io/secret/0281a566-f641-4a84-aa07-03f6e6845bc1-tls-certs\") pod \"0281a566-f641-4a84-aa07-03f6e6845bc1\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " Nov 02 18:01:22.483604 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.483590 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-tmp-dir\") pod \"0281a566-f641-4a84-aa07-03f6e6845bc1\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " Nov 02 18:01:22.483677 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.483626 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-dshm\") pod \"0281a566-f641-4a84-aa07-03f6e6845bc1\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " Nov 02 18:01:22.483752 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.483739 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-home\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-home\") pod \"0281a566-f641-4a84-aa07-03f6e6845bc1\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " Nov 02 18:01:22.483810 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.483768 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-model-cache\") pod \"0281a566-f641-4a84-aa07-03f6e6845bc1\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " Nov 02 18:01:22.483810 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.483793 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-tmp-dir" pod "0281a566-f641-4a84-aa07-03f6e6845bc1" (UID: "0281a566-f641-4a84-aa07-03f6e6845bc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:22.483986 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.483903 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0281a566-f641-4a84-aa07-03f6e6845bc1-kube-api-access-7vtk9\" (UniqueName: \"kubernetes.io/projected/0281a566-f641-4a84-aa07-03f6e6845bc1-kube-api-access-7vtk9\") pod \"0281a566-f641-4a84-aa07-03f6e6845bc1\" (UID: \"0281a566-f641-4a84-aa07-03f6e6845bc1\") " Nov 02 18:01:22.484160 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.484063 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-model-cache" pod "0281a566-f641-4a84-aa07-03f6e6845bc1" (UID: "0281a566-f641-4a84-aa07-03f6e6845bc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:22.484160 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.484073 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-home" pod "0281a566-f641-4a84-aa07-03f6e6845bc1" (UID: "0281a566-f641-4a84-aa07-03f6e6845bc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:22.484808 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.484780 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:01:22.485086 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.484813 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:01:22.485086 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.484828 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:01:22.486173 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.486136 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-dshm" pod "0281a566-f641-4a84-aa07-03f6e6845bc1" (UID: "0281a566-f641-4a84-aa07-03f6e6845bc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:22.486268 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.486177 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0281a566-f641-4a84-aa07-03f6e6845bc1-tls-certs" pod "0281a566-f641-4a84-aa07-03f6e6845bc1" (UID: "0281a566-f641-4a84-aa07-03f6e6845bc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:01:22.486416 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.486395 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0281a566-f641-4a84-aa07-03f6e6845bc1-kube-api-access-7vtk9" pod "0281a566-f641-4a84-aa07-03f6e6845bc1" (UID: "0281a566-f641-4a84-aa07-03f6e6845bc1"). InnerVolumeSpecName "kube-api-access-7vtk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:22.586651 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.586609 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0281a566-f641-4a84-aa07-03f6e6845bc1-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:01:22.586651 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.586654 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vtk9\" (UniqueName: \"kubernetes.io/projected/0281a566-f641-4a84-aa07-03f6e6845bc1-kube-api-access-7vtk9\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:01:22.587214 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:22.586671 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0281a566-f641-4a84-aa07-03f6e6845bc1-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:01:23.302195 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:23.302160 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" Nov 02 18:01:23.302371 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:23.302200 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf" event={"ID":"0281a566-f641-4a84-aa07-03f6e6845bc1","Type":"ContainerDied","Data":"d3d5185dbec40c1e305eabc9f957f1c6b08d41747e3019679af6428d66c2bea7"} Nov 02 18:01:23.302371 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:23.302255 4860 scope.go:122] "RemoveContainer" containerID="93e924edf3f941dbf758f5ce5e7ffab8586236d1ed6018e8ab1dfeb23fa305f8" Nov 02 18:01:23.361517 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:23.361478 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf"] Nov 02 18:01:23.373261 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:23.373231 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b913b2/test-model-b913b2-kserve-595f569bfc-jzwrf"] Nov 02 18:01:23.501712 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:23.501677 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0281a566-f641-4a84-aa07-03f6e6845bc1" path="/var/lib/kubelet/pods/0281a566-f641-4a84-aa07-03f6e6845bc1/volumes" Nov 02 18:01:30.631918 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:30.631876 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc"] Nov 02 18:01:30.632331 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:30.632088 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" podUID="7bba4e6a-f4d0-4840-8e6d-f229f330a336" containerName="istio-proxy" containerID="cri-o://36099368465b567ff1a182394d85986401c2a4e3953fb4eb9ee9f258b4324815" gracePeriod=30 Nov 02 18:01:35.928901 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:35.928872 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 18:01:36.066241 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.066204 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bba4e6a-f4d0-4840-8e6d-f229f330a336-kube-api-access-r2sgs\" (UniqueName: \"kubernetes.io/projected/7bba4e6a-f4d0-4840-8e6d-f229f330a336-kube-api-access-r2sgs\") pod \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " Nov 02 18:01:36.066436 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.066258 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-workload-certs\") pod \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " Nov 02 18:01:36.066436 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.066295 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-token\" (UniqueName: \"kubernetes.io/projected/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-token\") pod \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " Nov 02 18:01:36.066569 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.066446 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-data\") pod \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " Nov 02 18:01:36.066569 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.066503 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-workload-socket\") pod \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " Nov 02 18:01:36.066569 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.066538 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istiod-ca-cert\") pod \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " Nov 02 18:01:36.066738 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.066612 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-workload-certs" pod "7bba4e6a-f4d0-4840-8e6d-f229f330a336" (UID: "7bba4e6a-f4d0-4840-8e6d-f229f330a336"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:36.066738 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.066658 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-credential-socket\") pod \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " Nov 02 18:01:36.066905 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.066774 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-data" pod "7bba4e6a-f4d0-4840-8e6d-f229f330a336" (UID: "7bba4e6a-f4d0-4840-8e6d-f229f330a336"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:36.066905 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.066825 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-workload-socket" pod "7bba4e6a-f4d0-4840-8e6d-f229f330a336" (UID: "7bba4e6a-f4d0-4840-8e6d-f229f330a336"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:36.067029 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.066833 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-podinfo\") pod \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " Nov 02 18:01:36.067100 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.067025 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-envoy\") pod \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\" (UID: \"7bba4e6a-f4d0-4840-8e6d-f229f330a336\") " Nov 02 18:01:36.067100 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.067026 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-credential-socket" pod "7bba4e6a-f4d0-4840-8e6d-f229f330a336" (UID: "7bba4e6a-f4d0-4840-8e6d-f229f330a336"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:36.067100 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.067061 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istiod-ca-cert" pod "7bba4e6a-f4d0-4840-8e6d-f229f330a336" (UID: "7bba4e6a-f4d0-4840-8e6d-f229f330a336"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:01:36.067794 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.067769 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:01:36.067794 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.067795 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:01:36.067986 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.067808 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:01:36.067986 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.067822 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:01:36.067986 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.067834 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:01:36.069339 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.069313 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-podinfo" pod "7bba4e6a-f4d0-4840-8e6d-f229f330a336" (UID: "7bba4e6a-f4d0-4840-8e6d-f229f330a336"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:01:36.069465 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.069385 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-token" pod "7bba4e6a-f4d0-4840-8e6d-f229f330a336" (UID: "7bba4e6a-f4d0-4840-8e6d-f229f330a336"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:36.069465 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.069399 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-envoy" pod "7bba4e6a-f4d0-4840-8e6d-f229f330a336" (UID: "7bba4e6a-f4d0-4840-8e6d-f229f330a336"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:36.069465 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.069419 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bba4e6a-f4d0-4840-8e6d-f229f330a336-kube-api-access-r2sgs" pod "7bba4e6a-f4d0-4840-8e6d-f229f330a336" (UID: "7bba4e6a-f4d0-4840-8e6d-f229f330a336"). InnerVolumeSpecName "kube-api-access-r2sgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:36.170168 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.170082 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:01:36.170168 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.170113 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:01:36.170168 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.170123 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2sgs\" (UniqueName: \"kubernetes.io/projected/7bba4e6a-f4d0-4840-8e6d-f229f330a336-kube-api-access-r2sgs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:01:36.170168 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.170133 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bba4e6a-f4d0-4840-8e6d-f229f330a336-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:01:36.463535 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.463445 4860 generic.go:356] "Generic (PLEG): container finished" podID="7bba4e6a-f4d0-4840-8e6d-f229f330a336" containerID="36099368465b567ff1a182394d85986401c2a4e3953fb4eb9ee9f258b4324815" exitCode=0 Nov 02 18:01:36.463535 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.463497 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" Nov 02 18:01:36.463743 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.463532 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" event={"ID":"7bba4e6a-f4d0-4840-8e6d-f229f330a336","Type":"ContainerDied","Data":"36099368465b567ff1a182394d85986401c2a4e3953fb4eb9ee9f258b4324815"} Nov 02 18:01:36.463743 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.463616 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc" event={"ID":"7bba4e6a-f4d0-4840-8e6d-f229f330a336","Type":"ContainerDied","Data":"1b1bc243a9384b9b8bbe71858b9b43b43a29db30ef1596b8f38ad54dcf5b019d"} Nov 02 18:01:36.463743 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.463641 4860 scope.go:122] "RemoveContainer" containerID="36099368465b567ff1a182394d85986401c2a4e3953fb4eb9ee9f258b4324815" Nov 02 18:01:36.478189 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.478166 4860 scope.go:122] "RemoveContainer" containerID="36099368465b567ff1a182394d85986401c2a4e3953fb4eb9ee9f258b4324815" Nov 02 18:01:36.478535 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:01:36.478515 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36099368465b567ff1a182394d85986401c2a4e3953fb4eb9ee9f258b4324815\": container with ID starting with 36099368465b567ff1a182394d85986401c2a4e3953fb4eb9ee9f258b4324815 not found: ID does not exist" containerID="36099368465b567ff1a182394d85986401c2a4e3953fb4eb9ee9f258b4324815" Nov 02 18:01:36.478582 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.478548 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36099368465b567ff1a182394d85986401c2a4e3953fb4eb9ee9f258b4324815"} err="failed to get container status \"36099368465b567ff1a182394d85986401c2a4e3953fb4eb9ee9f258b4324815\": rpc error: code = NotFound desc = could not find container \"36099368465b567ff1a182394d85986401c2a4e3953fb4eb9ee9f258b4324815\": container with ID starting with 36099368465b567ff1a182394d85986401c2a4e3953fb4eb9ee9f258b4324815 not found: ID does not exist" Nov 02 18:01:36.501834 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.501794 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8"] Nov 02 18:01:36.503134 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.503108 4860 cpu_manager.go:414] "containerMap: removing container" podUID="7bba4e6a-f4d0-4840-8e6d-f229f330a336" containerName="istio-proxy" Nov 02 18:01:36.503134 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.503135 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bba4e6a-f4d0-4840-8e6d-f229f330a336" containerName="istio-proxy" Nov 02 18:01:36.503389 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.503183 4860 cpu_manager.go:414] "containerMap: removing container" podUID="0281a566-f641-4a84-aa07-03f6e6845bc1" containerName="main" Nov 02 18:01:36.503389 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.503191 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0281a566-f641-4a84-aa07-03f6e6845bc1" containerName="main" Nov 02 18:01:36.503389 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.503209 4860 cpu_manager.go:414] "containerMap: removing container" podUID="0ebc4571-d5c2-45f7-80de-5f79648303f4" containerName="istio-proxy" Nov 02 18:01:36.503389 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.503217 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebc4571-d5c2-45f7-80de-5f79648303f4" containerName="istio-proxy" Nov 02 18:01:36.503389 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.503278 4860 cpu_manager.go:414] "containerMap: removing container" podUID="da05c910-1ef7-47e6-a5bf-ead3b63064d8" containerName="istio-proxy" Nov 02 18:01:36.503389 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.503285 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da05c910-1ef7-47e6-a5bf-ead3b63064d8" containerName="istio-proxy" Nov 02 18:01:36.503988 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.503476 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0281a566-f641-4a84-aa07-03f6e6845bc1" containerName="main" Nov 02 18:01:36.503988 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.503491 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebc4571-d5c2-45f7-80de-5f79648303f4" containerName="istio-proxy" Nov 02 18:01:36.503988 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.503512 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="da05c910-1ef7-47e6-a5bf-ead3b63064d8" containerName="istio-proxy" Nov 02 18:01:36.503988 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.503525 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bba4e6a-f4d0-4840-8e6d-f229f330a336" containerName="istio-proxy" Nov 02 18:01:36.509004 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.508983 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.534624 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.534583 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8"] Nov 02 18:01:36.558513 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.558476 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc"] Nov 02 18:01:36.573324 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.573290 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-nn6qc"] Nov 02 18:01:36.594096 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.594063 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5fd3e0-openshift-default-dockercfg-kxlxh\"" Nov 02 18:01:36.678612 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.678571 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-data\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.678822 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.678719 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-workload-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.678822 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.678747 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qvgn\" (UniqueName: \"kubernetes.io/projected/1fb27095-a4ff-429a-91be-6f3e28872f18-kube-api-access-8qvgn\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.678979 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.678831 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-token\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.678979 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.678963 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-envoy\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.679092 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.679005 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-podinfo\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.679152 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.679125 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb27095-a4ff-429a-91be-6f3e28872f18-istiod-ca-cert\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.679212 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.679187 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-workload-certs\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.679421 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.679403 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-credential-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.781218 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.781125 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-credential-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.781218 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.781210 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-data\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.781462 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.781264 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-workload-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.781462 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.781283 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qvgn\" (UniqueName: \"kubernetes.io/projected/1fb27095-a4ff-429a-91be-6f3e28872f18-kube-api-access-8qvgn\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.781462 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.781333 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-token\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.781462 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.781363 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-envoy\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.781462 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.781394 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-podinfo\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.781744 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.781474 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb27095-a4ff-429a-91be-6f3e28872f18-istiod-ca-cert\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.781744 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.781531 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-workload-certs\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.781744 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.781610 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-credential-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.782168 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.782139 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-workload-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.782331 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.782311 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-data\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.782742 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.782716 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-workload-certs\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.782844 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.782756 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb27095-a4ff-429a-91be-6f3e28872f18-istiod-ca-cert\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.784247 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.784224 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-podinfo\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.784477 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.784451 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-envoy\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.805760 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.805735 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-token\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.805933 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.805908 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qvgn\" (UniqueName: \"kubernetes.io/projected/1fb27095-a4ff-429a-91be-6f3e28872f18-kube-api-access-8qvgn\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:36.826324 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:36.826287 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:37.158122 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:01:37.158082 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb27095_a4ff_429a_91be_6f3e28872f18.slice/crio-02157eed7e604d68113fa237257af267874fbc2b0b68302c4357f3e2d3f00481 WatchSource:0}: Error finding container 02157eed7e604d68113fa237257af267874fbc2b0b68302c4357f3e2d3f00481: Status 404 returned error can't find the container with id 02157eed7e604d68113fa237257af267874fbc2b0b68302c4357f3e2d3f00481 Nov 02 18:01:37.159267 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:37.159243 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8"] Nov 02 18:01:37.161522 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:37.161484 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:01:37.161629 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:37.161569 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:01:37.161694 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:37.161640 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:01:37.526795 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:37.526698 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bba4e6a-f4d0-4840-8e6d-f229f330a336" path="/var/lib/kubelet/pods/7bba4e6a-f4d0-4840-8e6d-f229f330a336/volumes" Nov 02 18:01:37.532900 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:37.527602 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" event={"ID":"1fb27095-a4ff-429a-91be-6f3e28872f18","Type":"ContainerStarted","Data":"40f12b1a26d758bb57b6b3844dba58f954012ae9ebe1a61242fb8fa54a56cd47"} Nov 02 18:01:37.532900 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:37.527683 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" event={"ID":"1fb27095-a4ff-429a-91be-6f3e28872f18","Type":"ContainerStarted","Data":"02157eed7e604d68113fa237257af267874fbc2b0b68302c4357f3e2d3f00481"} Nov 02 18:01:37.584715 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:37.584649 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" podStartSLOduration=1.58463377 podStartE2EDuration="1.58463377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:37.583175523 +0000 UTC m=+1916.204535301" watchObservedRunningTime="2026-11-02 18:01:37.58463377 +0000 UTC m=+1916.205993545" Nov 02 18:01:37.827374 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:37.827343 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:38.829484 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:38.829444 4860 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 18:01:38.830392 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:38.829507 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" podUID="1fb27095-a4ff-429a-91be-6f3e28872f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 18:01:38.839870 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:38.834561 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4"] Nov 02 18:01:38.847876 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:38.842181 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:38.897872 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:38.883956 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fbbbf8-openshift-default-dockercfg-b9pj2\"" Nov 02 18:01:38.906843 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:38.906797 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:38.907039 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:38.906903 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:38.907039 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:38.906947 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:38.907039 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:38.907023 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:38.907484 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:38.907097 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncmvp\" (UniqueName: \"kubernetes.io/projected/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-kube-api-access-ncmvp\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:38.907484 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:38.907138 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:38.907484 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:38.907217 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:38.907645 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:38.907566 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:38.907645 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:38.907636 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:38.907944 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:38.907904 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw"] Nov 02 18:01:38.934095 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:38.934059 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4"] Nov 02 18:01:38.934329 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:38.934297 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw"] Nov 02 18:01:38.934329 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:38.934201 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.009241 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.009208 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.009454 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.009347 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c12481ab-3025-40e3-a9a7-50b44df24d70-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.009454 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.009381 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.009571 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.009478 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9w9g\" (UniqueName: \"kubernetes.io/projected/c12481ab-3025-40e3-a9a7-50b44df24d70-kube-api-access-b9w9g\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.009571 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.009509 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.009688 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.009616 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.009688 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.009641 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.009688 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.009666 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.009838 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.009770 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:39.009973 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.009866 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:39.010030 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.009991 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:39.010084 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.010040 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:39.010084 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.010061 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:39.010186 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.010101 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:39.010186 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.010139 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmvp\" (UniqueName: \"kubernetes.io/projected/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-kube-api-access-ncmvp\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:39.010186 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.010161 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.010344 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.010187 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:39.010344 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.010230 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:39.010841 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.010819 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:39.012383 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.011409 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:39.012383 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.012008 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:39.012383 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.012047 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:39.012798 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.012773 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:39.013833 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.013809 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:39.014051 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.014034 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:39.034270 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.034240 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:39.036049 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.036019 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmvp\" (UniqueName: \"kubernetes.io/projected/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-kube-api-access-ncmvp\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:39.112471 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.112373 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.112629 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.112526 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.112629 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.112601 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c12481ab-3025-40e3-a9a7-50b44df24d70-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.112743 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.112642 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.112743 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.112693 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9w9g\" (UniqueName: \"kubernetes.io/projected/c12481ab-3025-40e3-a9a7-50b44df24d70-kube-api-access-b9w9g\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.112743 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.112727 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.112956 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.112815 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.112956 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.112864 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.112956 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.112883 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.112956 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.112906 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.113234 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.113209 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.113424 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.113402 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.113936 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.113669 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.113936 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.113810 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c12481ab-3025-40e3-a9a7-50b44df24d70-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.116007 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.115976 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.117090 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.117062 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.145390 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.144976 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9w9g\" (UniqueName: \"kubernetes.io/projected/c12481ab-3025-40e3-a9a7-50b44df24d70-kube-api-access-b9w9g\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.146398 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.146372 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.219875 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.219085 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:39.250739 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.250709 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:39.905123 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.904995 4860 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 18:01:39.906095 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:39.905973 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" podUID="1fb27095-a4ff-429a-91be-6f3e28872f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 18:01:40.029812 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:40.029781 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4"] Nov 02 18:01:40.081931 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:01:40.057121 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67e8cb6_152f_4cdd_a814_d1cfdbf99d96.slice/crio-5209dd3ae5514a823ed41826ba08565f19f4b17eaba3c28adeab4333e6efbc78 WatchSource:0}: Error finding container 5209dd3ae5514a823ed41826ba08565f19f4b17eaba3c28adeab4333e6efbc78: Status 404 returned error can't find the container with id 5209dd3ae5514a823ed41826ba08565f19f4b17eaba3c28adeab4333e6efbc78 Nov 02 18:01:40.096180 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:40.067011 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:01:40.096529 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:40.096492 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:01:40.096716 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:40.096692 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:01:40.315126 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:40.315097 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw"] Nov 02 18:01:40.320898 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:40.320839 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:01:40.320989 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:40.320950 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:01:40.321027 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:40.320988 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:01:40.826693 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:40.826656 4860 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 18:01:40.826910 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:40.826715 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" podUID="1fb27095-a4ff-429a-91be-6f3e28872f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 18:01:40.955605 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:40.955563 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" event={"ID":"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96","Type":"ContainerStarted","Data":"9b5f0c1e197f557ea4d9e9791cf957de49ace2a5cc7fb7bbcc0fcf335c7f0b64"} Nov 02 18:01:40.956871 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:40.956232 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" event={"ID":"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96","Type":"ContainerStarted","Data":"5209dd3ae5514a823ed41826ba08565f19f4b17eaba3c28adeab4333e6efbc78"} Nov 02 18:01:40.996713 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:40.992053 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" event={"ID":"c12481ab-3025-40e3-a9a7-50b44df24d70","Type":"ContainerStarted","Data":"0431ac7eb19a2ef02ca11ea864d1429253ec222e364a4510af75c05cf73bc61d"} Nov 02 18:01:40.996713 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:40.992126 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" event={"ID":"c12481ab-3025-40e3-a9a7-50b44df24d70","Type":"ContainerStarted","Data":"9852168cfd0f04238df8505eec3d064e0cecd9ef6ec15f2c31977ec42f9e982b"} Nov 02 18:01:41.198617 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:41.198494 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" podStartSLOduration=3.198472761 podStartE2EDuration="3.198472761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:41.191300947 +0000 UTC m=+1919.812660724" watchObservedRunningTime="2026-11-02 18:01:41.198472761 +0000 UTC m=+1919.819832537" Nov 02 18:01:41.239875 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:41.233834 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:41.239875 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:41.234390 4860 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 18:01:41.239875 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:41.234443 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" podUID="a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 18:01:41.283140 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:41.281469 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:41.283140 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:41.281516 4860 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 02 18:01:41.283140 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:41.281572 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" podUID="c12481ab-3025-40e3-a9a7-50b44df24d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 02 18:01:41.323875 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:41.310391 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" podStartSLOduration=3.310365712 podStartE2EDuration="3.310365712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:41.275337269 +0000 UTC m=+1919.896697050" watchObservedRunningTime="2026-11-02 18:01:41.310365712 +0000 UTC m=+1919.931725475" Nov 02 18:01:41.829969 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:41.829933 4860 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 18:01:41.830297 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:41.830224 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" podUID="1fb27095-a4ff-429a-91be-6f3e28872f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 18:01:42.222936 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:42.220250 4860 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 18:01:42.222936 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:42.220330 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" podUID="a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 18:01:42.251337 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:42.251297 4860 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 02 18:01:42.251543 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:42.251357 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" podUID="c12481ab-3025-40e3-a9a7-50b44df24d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 02 18:01:42.954903 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:42.949133 4860 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 18:01:42.954903 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:42.949195 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" podUID="1fb27095-a4ff-429a-91be-6f3e28872f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 18:01:43.123968 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:43.123917 4860 scope.go:122] "RemoveContainer" containerID="6b6c82d53d03ce3c28d7c2f39df0f2fdb88747cce90319002e49e4b86c77f9b5" Nov 02 18:01:43.220007 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:43.219711 4860 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 18:01:43.220007 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:43.219780 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" podUID="a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 18:01:43.262870 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:43.262442 4860 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 02 18:01:43.262870 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:43.262501 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" podUID="c12481ab-3025-40e3-a9a7-50b44df24d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 02 18:01:43.749344 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:43.749301 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:01:43.783823 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:43.783783 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:01:43.851089 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:43.838292 4860 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 18:01:43.851089 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:43.838343 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" podUID="1fb27095-a4ff-429a-91be-6f3e28872f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 18:01:43.969875 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:43.967336 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:01:44.016890 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:44.013695 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:01:44.049698 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:44.049650 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 18:01:44.161969 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:44.150303 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" podUID="23bfa30c-a0ac-4ec5-b12e-0d684c62ceff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.173:8000/ready\": context deadline exceeded" Nov 02 18:01:44.223052 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:44.222989 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:01:44.224671 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:44.224629 4860 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 18:01:44.233886 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:44.224882 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" podUID="a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 18:01:44.234209 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:44.227202 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:01:44.304968 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:44.304932 4860 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 02 18:01:44.305555 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:44.305524 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" podUID="c12481ab-3025-40e3-a9a7-50b44df24d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 02 18:01:44.647872 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:44.641308 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" podUID="23bfa30c-a0ac-4ec5-b12e-0d684c62ceff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.173:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:01:44.834892 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:44.829344 4860 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 18:01:44.834892 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:44.829418 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" podUID="1fb27095-a4ff-429a-91be-6f3e28872f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 18:01:45.221051 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:45.220265 4860 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 18:01:45.221051 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:45.220361 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" podUID="a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 18:01:45.251885 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:45.251834 4860 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 02 18:01:45.252100 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:45.251902 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" podUID="c12481ab-3025-40e3-a9a7-50b44df24d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 02 18:01:45.834561 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:45.834523 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:45.835605 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:45.835548 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:45.836103 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:45.836082 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:01:46.237974 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:46.232752 4860 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 18:01:46.238242 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:46.238200 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" podUID="a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 18:01:46.251282 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:46.251253 4860 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 02 18:01:46.251430 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:46.251301 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" podUID="c12481ab-3025-40e3-a9a7-50b44df24d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 02 18:01:47.219771 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:47.219734 4860 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 18:01:47.220240 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:47.219788 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" podUID="a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 18:01:47.251687 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:47.251653 4860 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 02 18:01:47.251828 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:47.251704 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" podUID="c12481ab-3025-40e3-a9a7-50b44df24d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 02 18:01:48.257144 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:48.257107 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:48.257629 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:48.257608 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:48.258374 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:48.258354 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:01:48.349121 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:48.349091 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:48.349445 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:48.349418 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:48.355443 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:01:48.355415 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:01:51.774730 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:01:51.774686 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:02:21.773077 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:02:21.772995 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:02:35.555915 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:35.555877 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8"] Nov 02 18:02:35.556428 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:35.556126 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" podUID="b6e9573f-76d6-4d96-a0a7-393ee6f63617" containerName="maas-api" containerID="cri-o://198fcb0cd815d01444d707f1a9b56f62f4c5144328b3822d59ca43dc1399fa86" gracePeriod=30 Nov 02 18:02:36.373868 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.373445 4860 generic.go:356] "Generic (PLEG): container finished" podID="b6e9573f-76d6-4d96-a0a7-393ee6f63617" containerID="198fcb0cd815d01444d707f1a9b56f62f4c5144328b3822d59ca43dc1399fa86" exitCode=0 Nov 02 18:02:36.374069 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.373871 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" event={"ID":"b6e9573f-76d6-4d96-a0a7-393ee6f63617","Type":"ContainerDied","Data":"198fcb0cd815d01444d707f1a9b56f62f4c5144328b3822d59ca43dc1399fa86"} Nov 02 18:02:36.392313 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.392159 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" Nov 02 18:02:36.437528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.437492 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e9573f-76d6-4d96-a0a7-393ee6f63617-kube-api-access-s8n5h\" (UniqueName: \"kubernetes.io/projected/b6e9573f-76d6-4d96-a0a7-393ee6f63617-kube-api-access-s8n5h\") pod \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\" (UID: \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\") " Nov 02 18:02:36.438057 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.438029 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6e9573f-76d6-4d96-a0a7-393ee6f63617-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6e9573f-76d6-4d96-a0a7-393ee6f63617-metrics-certs\") pod \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\" (UID: \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\") " Nov 02 18:02:36.438332 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.438311 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6e9573f-76d6-4d96-a0a7-393ee6f63617-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6e9573f-76d6-4d96-a0a7-393ee6f63617-maas-api-tls\") pod \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\" (UID: \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\") " Nov 02 18:02:36.438958 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.438936 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6e9573f-76d6-4d96-a0a7-393ee6f63617-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e9573f-76d6-4d96-a0a7-393ee6f63617-openshift-service-ca\") pod \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\" (UID: \"b6e9573f-76d6-4d96-a0a7-393ee6f63617\") " Nov 02 18:02:36.440824 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.440801 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e9573f-76d6-4d96-a0a7-393ee6f63617-openshift-service-ca" pod "b6e9573f-76d6-4d96-a0a7-393ee6f63617" (UID: "b6e9573f-76d6-4d96-a0a7-393ee6f63617"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:02:36.441791 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.441766 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e9573f-76d6-4d96-a0a7-393ee6f63617-kube-api-access-s8n5h" pod "b6e9573f-76d6-4d96-a0a7-393ee6f63617" (UID: "b6e9573f-76d6-4d96-a0a7-393ee6f63617"). InnerVolumeSpecName "kube-api-access-s8n5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:36.444615 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.444587 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e9573f-76d6-4d96-a0a7-393ee6f63617-maas-api-tls" pod "b6e9573f-76d6-4d96-a0a7-393ee6f63617" (UID: "b6e9573f-76d6-4d96-a0a7-393ee6f63617"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:02:36.487873 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.481447 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb"] Nov 02 18:02:36.487873 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.483952 4860 cpu_manager.go:414] "containerMap: removing container" podUID="b6e9573f-76d6-4d96-a0a7-393ee6f63617" containerName="maas-api" Nov 02 18:02:36.487873 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.483979 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6e9573f-76d6-4d96-a0a7-393ee6f63617" containerName="maas-api" Nov 02 18:02:36.487873 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.484253 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6e9573f-76d6-4d96-a0a7-393ee6f63617" containerName="maas-api" Nov 02 18:02:36.491219 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.490638 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.499982 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.499496 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e9573f-76d6-4d96-a0a7-393ee6f63617-metrics-certs" pod "b6e9573f-76d6-4d96-a0a7-393ee6f63617" (UID: "b6e9573f-76d6-4d96-a0a7-393ee6f63617"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:02:36.544910 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.542753 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.544910 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.542837 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.544910 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.542891 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.544910 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.542959 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.544910 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.543023 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.544910 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.543121 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.544910 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.543176 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.544910 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.543227 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.544910 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.543349 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzvqk\" (UniqueName: \"kubernetes.io/projected/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-kube-api-access-dzvqk\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.544910 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.543505 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6e9573f-76d6-4d96-a0a7-393ee6f63617-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:02:36.544910 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.543521 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6e9573f-76d6-4d96-a0a7-393ee6f63617-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:02:36.544910 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.543537 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e9573f-76d6-4d96-a0a7-393ee6f63617-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:02:36.544910 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.543551 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8n5h\" (UniqueName: \"kubernetes.io/projected/b6e9573f-76d6-4d96-a0a7-393ee6f63617-kube-api-access-s8n5h\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:02:36.549342 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.548803 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb"] Nov 02 18:02:36.645165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.645123 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzvqk\" (UniqueName: \"kubernetes.io/projected/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-kube-api-access-dzvqk\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.645598 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.645305 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.645598 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.645375 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.645598 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.645414 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.645598 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.645475 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.645598 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.645539 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.646064 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.645629 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.646064 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.645683 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.646064 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.645730 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.646318 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.646275 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.662910 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.647258 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.662910 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.648030 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.662910 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.648305 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.662910 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.659823 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.693022 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.684309 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.693265 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.693103 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzvqk\" (UniqueName: \"kubernetes.io/projected/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-kube-api-access-dzvqk\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.693687 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.693657 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.693826 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.693801 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:36.827874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:36.825398 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:37.225375 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:02:37.225343 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode31cdb01_9f71_4eae_ad26_6f1e19915ffd.slice/crio-ea9cf1ca6a488e802fa772efbc4eca2cb21a38b22ae36911f3f81706cfa45267 WatchSource:0}: Error finding container ea9cf1ca6a488e802fa772efbc4eca2cb21a38b22ae36911f3f81706cfa45267: Status 404 returned error can't find the container with id ea9cf1ca6a488e802fa772efbc4eca2cb21a38b22ae36911f3f81706cfa45267 Nov 02 18:02:37.227644 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:37.227616 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb"] Nov 02 18:02:37.228926 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:37.228884 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:02:37.229046 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:37.228979 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:02:37.229046 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:37.229029 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:02:37.386202 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:37.386171 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" Nov 02 18:02:37.386416 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:37.386166 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8" event={"ID":"b6e9573f-76d6-4d96-a0a7-393ee6f63617","Type":"ContainerDied","Data":"d04976470774be5102f4dcc4999ae1f228e1ec15f96ea4e43926bf0c03ba8600"} Nov 02 18:02:37.386416 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:37.386370 4860 scope.go:122] "RemoveContainer" containerID="198fcb0cd815d01444d707f1a9b56f62f4c5144328b3822d59ca43dc1399fa86" Nov 02 18:02:37.388380 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:37.388344 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" event={"ID":"e31cdb01-9f71-4eae-ad26-6f1e19915ffd","Type":"ContainerStarted","Data":"9da9acd413ede11a467a5cf900fff9debafc3b5dce5118d9af0863fbd69f080d"} Nov 02 18:02:37.388517 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:37.388410 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" event={"ID":"e31cdb01-9f71-4eae-ad26-6f1e19915ffd","Type":"ContainerStarted","Data":"ea9cf1ca6a488e802fa772efbc4eca2cb21a38b22ae36911f3f81706cfa45267"} Nov 02 18:02:37.457183 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:37.457130 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" podStartSLOduration=29.457115589 podStartE2EDuration="29.457115589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:37.455138196 +0000 UTC m=+1976.076497973" watchObservedRunningTime="2026-11-02 18:02:37.457115589 +0000 UTC m=+1976.078475364" Nov 02 18:02:37.503973 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:37.503898 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8"] Nov 02 18:02:37.511283 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:37.511248 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b913b2-5f9ffb46b-t6pc8"] Nov 02 18:02:37.826398 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:37.826365 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:38.844683 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:38.841342 4860 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 02 18:02:38.844683 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:38.841406 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" podUID="e31cdb01-9f71-4eae-ad26-6f1e19915ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 02 18:02:39.502555 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:39.502516 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6e9573f-76d6-4d96-a0a7-393ee6f63617" path="/var/lib/kubelet/pods/b6e9573f-76d6-4d96-a0a7-393ee6f63617/volumes" Nov 02 18:02:39.826477 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:39.826445 4860 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 02 18:02:39.827156 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:39.826511 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" podUID="e31cdb01-9f71-4eae-ad26-6f1e19915ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 02 18:02:40.827068 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:40.827017 4860 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 02 18:02:40.827635 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:40.827084 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" podUID="e31cdb01-9f71-4eae-ad26-6f1e19915ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 02 18:02:41.827106 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:41.827070 4860 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 02 18:02:41.827539 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:41.827132 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" podUID="e31cdb01-9f71-4eae-ad26-6f1e19915ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 02 18:02:42.827155 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:42.827116 4860 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 02 18:02:42.827565 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:42.827170 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" podUID="e31cdb01-9f71-4eae-ad26-6f1e19915ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 02 18:02:43.832472 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:43.832440 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:43.832987 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:43.832969 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:43.833821 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:43.833800 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:02:51.773328 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:02:51.773278 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:02:56.395040 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:56.395001 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h"] Nov 02 18:02:56.428782 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:56.420452 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h" Nov 02 18:02:56.442509 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:56.440527 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h"] Nov 02 18:02:56.466489 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:56.463287 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-557feb-dockercfg-nfpss\"" Nov 02 18:02:56.466489 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:56.463354 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-557feb\"" Nov 02 18:02:56.504388 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:56.504254 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82r8v\" (UniqueName: \"kubernetes.io/projected/f95eaf1d-668d-405f-bbc1-6be67fbe1fed-kube-api-access-82r8v\") pod \"payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h\" (UID: \"f95eaf1d-668d-405f-bbc1-6be67fbe1fed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h" Nov 02 18:02:56.504388 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:56.504348 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95eaf1d-668d-405f-bbc1-6be67fbe1fed-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h\" (UID: \"f95eaf1d-668d-405f-bbc1-6be67fbe1fed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h" Nov 02 18:02:56.607211 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:56.607171 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82r8v\" (UniqueName: \"kubernetes.io/projected/f95eaf1d-668d-405f-bbc1-6be67fbe1fed-kube-api-access-82r8v\") pod \"payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h\" (UID: \"f95eaf1d-668d-405f-bbc1-6be67fbe1fed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h" Nov 02 18:02:56.607421 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:56.607266 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95eaf1d-668d-405f-bbc1-6be67fbe1fed-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h\" (UID: \"f95eaf1d-668d-405f-bbc1-6be67fbe1fed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h" Nov 02 18:02:56.608045 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:56.608017 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95eaf1d-668d-405f-bbc1-6be67fbe1fed-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h\" (UID: \"f95eaf1d-668d-405f-bbc1-6be67fbe1fed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h" Nov 02 18:02:56.634033 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:56.634001 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82r8v\" (UniqueName: \"kubernetes.io/projected/f95eaf1d-668d-405f-bbc1-6be67fbe1fed-kube-api-access-82r8v\") pod \"payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h\" (UID: \"f95eaf1d-668d-405f-bbc1-6be67fbe1fed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h" Nov 02 18:02:56.830688 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:56.830655 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h" Nov 02 18:02:57.151823 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:02:57.151787 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95eaf1d_668d_405f_bbc1_6be67fbe1fed.slice/crio-085b79d247b534d38e78ae0baf83a1bed4f8908a574c03f8913a042e96ef1d89 WatchSource:0}: Error finding container 085b79d247b534d38e78ae0baf83a1bed4f8908a574c03f8913a042e96ef1d89: Status 404 returned error can't find the container with id 085b79d247b534d38e78ae0baf83a1bed4f8908a574c03f8913a042e96ef1d89 Nov 02 18:02:57.152577 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:57.152549 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h"] Nov 02 18:02:57.671687 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:57.671630 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h" event={"ID":"f95eaf1d-668d-405f-bbc1-6be67fbe1fed","Type":"ContainerStarted","Data":"0928ae144f63d074479af700db10032736cc9799383bf76cc273845df6f52d8e"} Nov 02 18:02:57.672123 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:57.671827 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h" Nov 02 18:02:57.672123 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:57.671879 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h" event={"ID":"f95eaf1d-668d-405f-bbc1-6be67fbe1fed","Type":"ContainerStarted","Data":"085b79d247b534d38e78ae0baf83a1bed4f8908a574c03f8913a042e96ef1d89"} Nov 02 18:02:57.718071 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:02:57.718017 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h" podStartSLOduration=1.7180005980000002 podStartE2EDuration="1.718000598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:57.712150858 +0000 UTC m=+1996.333510634" watchObservedRunningTime="2026-11-02 18:02:57.718000598 +0000 UTC m=+1996.339360375" Nov 02 18:03:08.715885 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:08.715318 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h" Nov 02 18:03:11.688979 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:11.688926 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:11.746973 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:11.746927 4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:11.809990 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:11.809942 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:11.821934 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:11.821888 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:11.863091 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:11.863032 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:11.881041 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:11.880995 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 02 18:03:11.980807 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:11.980721 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:12.036954 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:12.036910 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 02 18:03:21.773158 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:03:21.773119 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:03:37.816829 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:37.816789 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/odh-model-controller-769875db56-27r8m"] Nov 02 18:03:38.274161 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:38.274066 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/odh-model-controller-769875db56-27r8m" podUID="f2fa7be0-9911-423c-adad-e88db47815a0" containerName="manager" containerID="cri-o://c8bb9d5b3ecaa1eec23962624837d3c024ab202620f4dd6dead4f1a523e37d6f" gracePeriod=10 Nov 02 18:03:38.591271 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:38.591242 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-27r8m" Nov 02 18:03:38.708609 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:38.708577 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2fa7be0-9911-423c-adad-e88db47815a0-cert\" (UniqueName: \"kubernetes.io/secret/f2fa7be0-9911-423c-adad-e88db47815a0-cert\") pod \"f2fa7be0-9911-423c-adad-e88db47815a0\" (UID: \"f2fa7be0-9911-423c-adad-e88db47815a0\") " Nov 02 18:03:38.708784 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:38.708714 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2fa7be0-9911-423c-adad-e88db47815a0-kube-api-access-tswzj\" (UniqueName: \"kubernetes.io/projected/f2fa7be0-9911-423c-adad-e88db47815a0-kube-api-access-tswzj\") pod \"f2fa7be0-9911-423c-adad-e88db47815a0\" (UID: \"f2fa7be0-9911-423c-adad-e88db47815a0\") " Nov 02 18:03:38.708887 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:38.708830 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2fa7be0-9911-423c-adad-e88db47815a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2fa7be0-9911-423c-adad-e88db47815a0-metrics-certs\") pod \"f2fa7be0-9911-423c-adad-e88db47815a0\" (UID: \"f2fa7be0-9911-423c-adad-e88db47815a0\") " Nov 02 18:03:38.710894 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:38.710824 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fa7be0-9911-423c-adad-e88db47815a0-cert" pod "f2fa7be0-9911-423c-adad-e88db47815a0" (UID: "f2fa7be0-9911-423c-adad-e88db47815a0"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:03:38.710894 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:38.710842 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fa7be0-9911-423c-adad-e88db47815a0-metrics-certs" pod "f2fa7be0-9911-423c-adad-e88db47815a0" (UID: "f2fa7be0-9911-423c-adad-e88db47815a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:03:38.711058 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:38.710976 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fa7be0-9911-423c-adad-e88db47815a0-kube-api-access-tswzj" pod "f2fa7be0-9911-423c-adad-e88db47815a0" (UID: "f2fa7be0-9911-423c-adad-e88db47815a0"). InnerVolumeSpecName "kube-api-access-tswzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:38.811228 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:38.811190 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2fa7be0-9911-423c-adad-e88db47815a0-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:03:38.811228 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:38.811219 4860 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2fa7be0-9911-423c-adad-e88db47815a0-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:03:38.811228 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:38.811228 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tswzj\" (UniqueName: \"kubernetes.io/projected/f2fa7be0-9911-423c-adad-e88db47815a0-kube-api-access-tswzj\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:03:39.284110 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:39.284009 4860 generic.go:356] "Generic (PLEG): container finished" podID="f2fa7be0-9911-423c-adad-e88db47815a0" containerID="c8bb9d5b3ecaa1eec23962624837d3c024ab202620f4dd6dead4f1a523e37d6f" exitCode=0 Nov 02 18:03:39.284110 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:39.284066 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-27r8m" Nov 02 18:03:39.284110 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:39.284100 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-27r8m" event={"ID":"f2fa7be0-9911-423c-adad-e88db47815a0","Type":"ContainerDied","Data":"c8bb9d5b3ecaa1eec23962624837d3c024ab202620f4dd6dead4f1a523e37d6f"} Nov 02 18:03:39.284636 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:39.284198 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-27r8m" event={"ID":"f2fa7be0-9911-423c-adad-e88db47815a0","Type":"ContainerDied","Data":"70aea73a25a6b08c16b7459cdd6c8a73fed95926023b51ed55bc0a797b8dd0d8"} Nov 02 18:03:39.284636 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:39.284228 4860 scope.go:122] "RemoveContainer" containerID="c8bb9d5b3ecaa1eec23962624837d3c024ab202620f4dd6dead4f1a523e37d6f" Nov 02 18:03:39.303470 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:39.303450 4860 scope.go:122] "RemoveContainer" containerID="c8bb9d5b3ecaa1eec23962624837d3c024ab202620f4dd6dead4f1a523e37d6f" Nov 02 18:03:39.303729 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:03:39.303710 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8bb9d5b3ecaa1eec23962624837d3c024ab202620f4dd6dead4f1a523e37d6f\": container with ID starting with c8bb9d5b3ecaa1eec23962624837d3c024ab202620f4dd6dead4f1a523e37d6f not found: ID does not exist" containerID="c8bb9d5b3ecaa1eec23962624837d3c024ab202620f4dd6dead4f1a523e37d6f" Nov 02 18:03:39.303793 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:39.303737 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8bb9d5b3ecaa1eec23962624837d3c024ab202620f4dd6dead4f1a523e37d6f"} err="failed to get container status \"c8bb9d5b3ecaa1eec23962624837d3c024ab202620f4dd6dead4f1a523e37d6f\": rpc error: code = NotFound desc = could not find container \"c8bb9d5b3ecaa1eec23962624837d3c024ab202620f4dd6dead4f1a523e37d6f\": container with ID starting with c8bb9d5b3ecaa1eec23962624837d3c024ab202620f4dd6dead4f1a523e37d6f not found: ID does not exist" Nov 02 18:03:39.340556 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:39.340528 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/odh-model-controller-769875db56-27r8m"] Nov 02 18:03:39.352638 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:39.352613 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/odh-model-controller-769875db56-27r8m"] Nov 02 18:03:39.501502 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:39.501461 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2fa7be0-9911-423c-adad-e88db47815a0" path="/var/lib/kubelet/pods/f2fa7be0-9911-423c-adad-e88db47815a0/volumes" Nov 02 18:03:49.279401 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.279364 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-kqv5l"] Nov 02 18:03:49.280295 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.280277 4860 cpu_manager.go:414] "containerMap: removing container" podUID="f2fa7be0-9911-423c-adad-e88db47815a0" containerName="manager" Nov 02 18:03:49.280355 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.280296 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2fa7be0-9911-423c-adad-e88db47815a0" containerName="manager" Nov 02 18:03:49.280453 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.280444 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2fa7be0-9911-423c-adad-e88db47815a0" containerName="manager" Nov 02 18:03:49.286075 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.286055 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-kqv5l" Nov 02 18:03:49.303784 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.303758 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-kqv5l"] Nov 02 18:03:49.309347 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.309317 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 02 18:03:49.313514 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.313492 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 02 18:03:49.313665 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.313639 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-4xbwb\"" Nov 02 18:03:49.412693 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.412659 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a1cac0c-2637-47b7-a9c4-d81bca026068-cert\") pod \"odh-model-controller-769875db56-kqv5l\" (UID: \"3a1cac0c-2637-47b7-a9c4-d81bca026068\") " pod="opendatahub/odh-model-controller-769875db56-kqv5l" Nov 02 18:03:49.412988 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.412942 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4pgs\" (UniqueName: \"kubernetes.io/projected/3a1cac0c-2637-47b7-a9c4-d81bca026068-kube-api-access-k4pgs\") pod \"odh-model-controller-769875db56-kqv5l\" (UID: \"3a1cac0c-2637-47b7-a9c4-d81bca026068\") " pod="opendatahub/odh-model-controller-769875db56-kqv5l" Nov 02 18:03:49.413137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.413120 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a1cac0c-2637-47b7-a9c4-d81bca026068-metrics-certs\") pod \"odh-model-controller-769875db56-kqv5l\" (UID: \"3a1cac0c-2637-47b7-a9c4-d81bca026068\") " pod="opendatahub/odh-model-controller-769875db56-kqv5l" Nov 02 18:03:49.514195 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.514161 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a1cac0c-2637-47b7-a9c4-d81bca026068-metrics-certs\") pod \"odh-model-controller-769875db56-kqv5l\" (UID: \"3a1cac0c-2637-47b7-a9c4-d81bca026068\") " pod="opendatahub/odh-model-controller-769875db56-kqv5l" Nov 02 18:03:49.514396 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.514246 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a1cac0c-2637-47b7-a9c4-d81bca026068-cert\") pod \"odh-model-controller-769875db56-kqv5l\" (UID: \"3a1cac0c-2637-47b7-a9c4-d81bca026068\") " pod="opendatahub/odh-model-controller-769875db56-kqv5l" Nov 02 18:03:49.514464 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.514413 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4pgs\" (UniqueName: \"kubernetes.io/projected/3a1cac0c-2637-47b7-a9c4-d81bca026068-kube-api-access-k4pgs\") pod \"odh-model-controller-769875db56-kqv5l\" (UID: \"3a1cac0c-2637-47b7-a9c4-d81bca026068\") " pod="opendatahub/odh-model-controller-769875db56-kqv5l" Nov 02 18:03:49.516698 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.516672 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a1cac0c-2637-47b7-a9c4-d81bca026068-cert\") pod \"odh-model-controller-769875db56-kqv5l\" (UID: \"3a1cac0c-2637-47b7-a9c4-d81bca026068\") " pod="opendatahub/odh-model-controller-769875db56-kqv5l" Nov 02 18:03:49.516817 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.516773 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a1cac0c-2637-47b7-a9c4-d81bca026068-metrics-certs\") pod \"odh-model-controller-769875db56-kqv5l\" (UID: \"3a1cac0c-2637-47b7-a9c4-d81bca026068\") " pod="opendatahub/odh-model-controller-769875db56-kqv5l" Nov 02 18:03:49.538302 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.538233 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4pgs\" (UniqueName: \"kubernetes.io/projected/3a1cac0c-2637-47b7-a9c4-d81bca026068-kube-api-access-k4pgs\") pod \"odh-model-controller-769875db56-kqv5l\" (UID: \"3a1cac0c-2637-47b7-a9c4-d81bca026068\") " pod="opendatahub/odh-model-controller-769875db56-kqv5l" Nov 02 18:03:49.603000 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.602968 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-kqv5l" Nov 02 18:03:49.920304 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:03:49.920272 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1cac0c_2637_47b7_a9c4_d81bca026068.slice/crio-384ed11cbf2fb8a8dd1622d4cf446bfc083d6f4bd3cea39a217c4c18936826d9 WatchSource:0}: Error finding container 384ed11cbf2fb8a8dd1622d4cf446bfc083d6f4bd3cea39a217c4c18936826d9: Status 404 returned error can't find the container with id 384ed11cbf2fb8a8dd1622d4cf446bfc083d6f4bd3cea39a217c4c18936826d9 Nov 02 18:03:49.921139 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.921108 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-kqv5l"] Nov 02 18:03:49.921832 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:49.921813 4860 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:03:50.424036 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:50.423997 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-kqv5l" event={"ID":"3a1cac0c-2637-47b7-a9c4-d81bca026068","Type":"ContainerStarted","Data":"384ed11cbf2fb8a8dd1622d4cf446bfc083d6f4bd3cea39a217c4c18936826d9"} Nov 02 18:03:51.434950 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:51.434903 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-kqv5l" event={"ID":"3a1cac0c-2637-47b7-a9c4-d81bca026068","Type":"ContainerStarted","Data":"ff3368e3e99ff78ec5f9508ddb2da61d4a79b1076a938cddc72ed96a17006ea4"} Nov 02 18:03:51.435381 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:51.434996 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-kqv5l" Nov 02 18:03:51.482546 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:03:51.482491 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-kqv5l" podStartSLOduration=2.027989357 podStartE2EDuration="2.48247637s" totalImagesPullingTime="454.487013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:51.478943178 +0000 UTC m=+2050.100302955" watchObservedRunningTime="2026-11-02 18:03:51.48247637 +0000 UTC m=+2050.103836147" Nov 02 18:03:51.776655 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:03:51.776561 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:04:02.447917 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:04:02.447881 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-kqv5l" Nov 02 18:04:14.838944 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:04:14.838906 4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vs4r4_daf51d1a-a65f-4fad-8a82-966b8bac7ba4/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:21.773214 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:04:21.773170 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:04:41.979008 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:04:41.978981 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 18:04:41.991410 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:04:41.991385 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 18:04:41.993279 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:04:41.993258 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 18:04:42.004818 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:04:42.004796 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 18:04:51.773394 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:04:51.773357 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:05:04.148725 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:04.148268 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678"] Nov 02 18:05:04.159298 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:04.159217 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678" Nov 02 18:05:04.192726 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:04.192673 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678"] Nov 02 18:05:04.193293 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:04.193256 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-227939\"" Nov 02 18:05:04.193653 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:04.193564 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-227939-dockercfg-4rtbn\"" Nov 02 18:05:04.344887 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:04.337611 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e2db1f-bd9c-46f0-b913-9533fcf95a4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678\" (UID: \"e1e2db1f-bd9c-46f0-b913-9533fcf95a4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678" Nov 02 18:05:04.345272 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:04.345238 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7z4s\" (UniqueName: \"kubernetes.io/projected/e1e2db1f-bd9c-46f0-b913-9533fcf95a4d-kube-api-access-s7z4s\") pod \"payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678\" (UID: \"e1e2db1f-bd9c-46f0-b913-9533fcf95a4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678" Nov 02 18:05:04.480552 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:04.480418 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e2db1f-bd9c-46f0-b913-9533fcf95a4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678\" (UID: \"e1e2db1f-bd9c-46f0-b913-9533fcf95a4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678" Nov 02 18:05:04.480552 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:04.480535 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7z4s\" (UniqueName: \"kubernetes.io/projected/e1e2db1f-bd9c-46f0-b913-9533fcf95a4d-kube-api-access-s7z4s\") pod \"payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678\" (UID: \"e1e2db1f-bd9c-46f0-b913-9533fcf95a4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678" Nov 02 18:05:04.503204 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:04.503167 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e2db1f-bd9c-46f0-b913-9533fcf95a4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678\" (UID: \"e1e2db1f-bd9c-46f0-b913-9533fcf95a4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678" Nov 02 18:05:04.538835 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:04.538378 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7z4s\" (UniqueName: \"kubernetes.io/projected/e1e2db1f-bd9c-46f0-b913-9533fcf95a4d-kube-api-access-s7z4s\") pod \"payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678\" (UID: \"e1e2db1f-bd9c-46f0-b913-9533fcf95a4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678" Nov 02 18:05:04.794970 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:04.794869 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678" Nov 02 18:05:05.173517 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:05:05.173484 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e2db1f_bd9c_46f0_b913_9533fcf95a4d.slice/crio-db341a5bea5df22935f0045a692cac0ea482a73f182f65fb10d702282ad2f9ff WatchSource:0}: Error finding container db341a5bea5df22935f0045a692cac0ea482a73f182f65fb10d702282ad2f9ff: Status 404 returned error can't find the container with id db341a5bea5df22935f0045a692cac0ea482a73f182f65fb10d702282ad2f9ff Nov 02 18:05:05.177179 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:05.177146 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678"] Nov 02 18:05:05.545549 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:05.545442 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678" event={"ID":"e1e2db1f-bd9c-46f0-b913-9533fcf95a4d","Type":"ContainerStarted","Data":"6c9a84e669cf98e98d0b3e37beced94c7b4efa2976da541f540e241b83a982bf"} Nov 02 18:05:05.545549 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:05.545515 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678" event={"ID":"e1e2db1f-bd9c-46f0-b913-9533fcf95a4d","Type":"ContainerStarted","Data":"db341a5bea5df22935f0045a692cac0ea482a73f182f65fb10d702282ad2f9ff"} Nov 02 18:05:05.545832 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:05.545569 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678" Nov 02 18:05:05.594799 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:05.594742 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678" podStartSLOduration=1.594727641 podStartE2EDuration="1.594727641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:05.589240424 +0000 UTC m=+2124.210600199" watchObservedRunningTime="2026-11-02 18:05:05.594727641 +0000 UTC m=+2124.216087414" Nov 02 18:05:16.565577 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:16.565547 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678" Nov 02 18:05:21.776402 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:05:21.776359 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:05:29.433895 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:29.433842 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h"] Nov 02 18:05:38.737156 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:38.737111 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf"] Nov 02 18:05:42.147751 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:42.147711 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w"] Nov 02 18:05:44.461839 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.461769 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" podUID="48350106-29a0-4a22-a995-203997f1d909" containerName="main" containerID="cri-o://b98a4fce807a65fb27d604360bab90cfc1da48ebc82d039421e83be7228991e8" gracePeriod=45 Nov 02 18:05:44.799326 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.799293 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 18:05:44.835138 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.835100 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-home\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-home\") pod \"48350106-29a0-4a22-a995-203997f1d909\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " Nov 02 18:05:44.835300 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.835253 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48350106-29a0-4a22-a995-203997f1d909-tls-certs\" (UniqueName: \"kubernetes.io/secret/48350106-29a0-4a22-a995-203997f1d909-tls-certs\") pod \"48350106-29a0-4a22-a995-203997f1d909\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " Nov 02 18:05:44.835372 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.835327 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-model-cache\") pod \"48350106-29a0-4a22-a995-203997f1d909\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " Nov 02 18:05:44.835372 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.835351 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-dshm\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-dshm\") pod \"48350106-29a0-4a22-a995-203997f1d909\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " Nov 02 18:05:44.835485 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.835401 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-home" pod "48350106-29a0-4a22-a995-203997f1d909" (UID: "48350106-29a0-4a22-a995-203997f1d909"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:44.835548 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.835483 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-tmp-dir\") pod \"48350106-29a0-4a22-a995-203997f1d909\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " Nov 02 18:05:44.835606 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.835562 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48350106-29a0-4a22-a995-203997f1d909-kube-api-access-g5bgp\" (UniqueName: \"kubernetes.io/projected/48350106-29a0-4a22-a995-203997f1d909-kube-api-access-g5bgp\") pod \"48350106-29a0-4a22-a995-203997f1d909\" (UID: \"48350106-29a0-4a22-a995-203997f1d909\") " Nov 02 18:05:44.835688 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.835644 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-model-cache" pod "48350106-29a0-4a22-a995-203997f1d909" (UID: "48350106-29a0-4a22-a995-203997f1d909"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:44.835743 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.835715 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-tmp-dir" pod "48350106-29a0-4a22-a995-203997f1d909" (UID: "48350106-29a0-4a22-a995-203997f1d909"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:44.836583 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.836557 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:05:44.836583 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.836582 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:05:44.836724 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.836597 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:05:44.838091 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.838063 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48350106-29a0-4a22-a995-203997f1d909-tls-certs" pod "48350106-29a0-4a22-a995-203997f1d909" (UID: "48350106-29a0-4a22-a995-203997f1d909"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:44.838315 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.838292 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-dshm" pod "48350106-29a0-4a22-a995-203997f1d909" (UID: "48350106-29a0-4a22-a995-203997f1d909"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:44.838315 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.838351 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48350106-29a0-4a22-a995-203997f1d909-kube-api-access-g5bgp" pod "48350106-29a0-4a22-a995-203997f1d909" (UID: "48350106-29a0-4a22-a995-203997f1d909"). InnerVolumeSpecName "kube-api-access-g5bgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:44.938210 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.938173 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5bgp\" (UniqueName: \"kubernetes.io/projected/48350106-29a0-4a22-a995-203997f1d909-kube-api-access-g5bgp\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:05:44.938210 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.938205 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48350106-29a0-4a22-a995-203997f1d909-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:05:44.938210 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:44.938214 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48350106-29a0-4a22-a995-203997f1d909-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:05:45.202156 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:45.202114 4860 generic.go:356] "Generic (PLEG): container finished" podID="48350106-29a0-4a22-a995-203997f1d909" containerID="b98a4fce807a65fb27d604360bab90cfc1da48ebc82d039421e83be7228991e8" exitCode=0 Nov 02 18:05:45.202362 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:45.202261 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" event={"ID":"48350106-29a0-4a22-a995-203997f1d909","Type":"ContainerDied","Data":"b98a4fce807a65fb27d604360bab90cfc1da48ebc82d039421e83be7228991e8"} Nov 02 18:05:45.202362 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:45.202315 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" event={"ID":"48350106-29a0-4a22-a995-203997f1d909","Type":"ContainerDied","Data":"a34ddaa22c7195e13674daf34ef85feab03f3e8e0cc370809e1762f1652413dd"} Nov 02 18:05:45.202362 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:45.202345 4860 scope.go:122] "RemoveContainer" containerID="b98a4fce807a65fb27d604360bab90cfc1da48ebc82d039421e83be7228991e8" Nov 02 18:05:45.202898 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:45.202836 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h" Nov 02 18:05:45.215733 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:45.215716 4860 scope.go:122] "RemoveContainer" containerID="b98a4fce807a65fb27d604360bab90cfc1da48ebc82d039421e83be7228991e8" Nov 02 18:05:45.216041 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:05:45.216020 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98a4fce807a65fb27d604360bab90cfc1da48ebc82d039421e83be7228991e8\": container with ID starting with b98a4fce807a65fb27d604360bab90cfc1da48ebc82d039421e83be7228991e8 not found: ID does not exist" containerID="b98a4fce807a65fb27d604360bab90cfc1da48ebc82d039421e83be7228991e8" Nov 02 18:05:45.216088 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:45.216048 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98a4fce807a65fb27d604360bab90cfc1da48ebc82d039421e83be7228991e8"} err="failed to get container status \"b98a4fce807a65fb27d604360bab90cfc1da48ebc82d039421e83be7228991e8\": rpc error: code = NotFound desc = could not find container \"b98a4fce807a65fb27d604360bab90cfc1da48ebc82d039421e83be7228991e8\": container with ID starting with b98a4fce807a65fb27d604360bab90cfc1da48ebc82d039421e83be7228991e8 not found: ID does not exist" Nov 02 18:05:45.259092 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:45.259063 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h"] Nov 02 18:05:45.270534 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:45.270503 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a5c1da/test-model-a5c1da-kserve-68ffbb7dc7-q9t2h"] Nov 02 18:05:45.502893 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:45.502791 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48350106-29a0-4a22-a995-203997f1d909" path="/var/lib/kubelet/pods/48350106-29a0-4a22-a995-203997f1d909/volumes" Nov 02 18:05:51.772258 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:05:51.772213 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:05:53.775908 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:53.775835 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" podUID="d26e69c6-ae7f-4fcd-895f-4dcd81062d67" containerName="main" containerID="cri-o://7aed4f05b563fc21f2372957a370390691fa5d1caf9912ce62057c0db3f5b3de" gracePeriod=45 Nov 02 18:05:54.187216 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.187189 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 18:05:54.298525 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.298438 4860 generic.go:356] "Generic (PLEG): container finished" podID="d26e69c6-ae7f-4fcd-895f-4dcd81062d67" containerID="7aed4f05b563fc21f2372957a370390691fa5d1caf9912ce62057c0db3f5b3de" exitCode=0 Nov 02 18:05:54.298525 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.298485 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" Nov 02 18:05:54.298525 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.298513 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" event={"ID":"d26e69c6-ae7f-4fcd-895f-4dcd81062d67","Type":"ContainerDied","Data":"7aed4f05b563fc21f2372957a370390691fa5d1caf9912ce62057c0db3f5b3de"} Nov 02 18:05:54.298817 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.298558 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf" event={"ID":"d26e69c6-ae7f-4fcd-895f-4dcd81062d67","Type":"ContainerDied","Data":"0b6fa64ccbfaeb38d95126731fe5e3ebdd480640901163c01574827daa5e3a1f"} Nov 02 18:05:54.298817 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.298574 4860 scope.go:122] "RemoveContainer" containerID="7aed4f05b563fc21f2372957a370390691fa5d1caf9912ce62057c0db3f5b3de" Nov 02 18:05:54.309860 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.309825 4860 scope.go:122] "RemoveContainer" containerID="7aed4f05b563fc21f2372957a370390691fa5d1caf9912ce62057c0db3f5b3de" Nov 02 18:05:54.310162 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:05:54.310138 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aed4f05b563fc21f2372957a370390691fa5d1caf9912ce62057c0db3f5b3de\": container with ID starting with 7aed4f05b563fc21f2372957a370390691fa5d1caf9912ce62057c0db3f5b3de not found: ID does not exist" containerID="7aed4f05b563fc21f2372957a370390691fa5d1caf9912ce62057c0db3f5b3de" Nov 02 18:05:54.310256 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.310169 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aed4f05b563fc21f2372957a370390691fa5d1caf9912ce62057c0db3f5b3de"} err="failed to get container status \"7aed4f05b563fc21f2372957a370390691fa5d1caf9912ce62057c0db3f5b3de\": rpc error: code = NotFound desc = could not find container \"7aed4f05b563fc21f2372957a370390691fa5d1caf9912ce62057c0db3f5b3de\": container with ID starting with 7aed4f05b563fc21f2372957a370390691fa5d1caf9912ce62057c0db3f5b3de not found: ID does not exist" Nov 02 18:05:54.317431 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.317407 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-dshm\") pod \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " Nov 02 18:05:54.317523 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.317461 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-tls-certs\" (UniqueName: \"kubernetes.io/secret/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-tls-certs\") pod \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " Nov 02 18:05:54.317523 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.317492 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-tmp-dir\") pod \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " Nov 02 18:05:54.317624 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.317608 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-model-cache\") pod \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " Nov 02 18:05:54.317735 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.317719 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-home\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-home\") pod \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " Nov 02 18:05:54.317817 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.317752 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-kube-api-access-b4w8g\" (UniqueName: \"kubernetes.io/projected/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-kube-api-access-b4w8g\") pod \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\" (UID: \"d26e69c6-ae7f-4fcd-895f-4dcd81062d67\") " Nov 02 18:05:54.317889 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.317827 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-tmp-dir" pod "d26e69c6-ae7f-4fcd-895f-4dcd81062d67" (UID: "d26e69c6-ae7f-4fcd-895f-4dcd81062d67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:54.317889 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.317844 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-model-cache" pod "d26e69c6-ae7f-4fcd-895f-4dcd81062d67" (UID: "d26e69c6-ae7f-4fcd-895f-4dcd81062d67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:54.318015 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.317959 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-home" pod "d26e69c6-ae7f-4fcd-895f-4dcd81062d67" (UID: "d26e69c6-ae7f-4fcd-895f-4dcd81062d67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:54.318476 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.318458 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:05:54.318547 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.318480 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:05:54.318547 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.318489 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:05:54.320089 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.320061 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-kube-api-access-b4w8g" pod "d26e69c6-ae7f-4fcd-895f-4dcd81062d67" (UID: "d26e69c6-ae7f-4fcd-895f-4dcd81062d67"). InnerVolumeSpecName "kube-api-access-b4w8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:54.320181 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.320149 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-dshm" pod "d26e69c6-ae7f-4fcd-895f-4dcd81062d67" (UID: "d26e69c6-ae7f-4fcd-895f-4dcd81062d67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:54.320181 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.320156 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-tls-certs" pod "d26e69c6-ae7f-4fcd-895f-4dcd81062d67" (UID: "d26e69c6-ae7f-4fcd-895f-4dcd81062d67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:54.420380 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.420340 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:05:54.420380 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.420369 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:05:54.420380 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.420380 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4w8g\" (UniqueName: \"kubernetes.io/projected/d26e69c6-ae7f-4fcd-895f-4dcd81062d67-kube-api-access-b4w8g\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:05:54.648717 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.648686 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf"] Nov 02 18:05:54.654229 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:54.654197 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b2fc9f/test-model-b2fc9f-kserve-6f9cb5f967-8cbzf"] Nov 02 18:05:55.502399 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:55.502366 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26e69c6-ae7f-4fcd-895f-4dcd81062d67" path="/var/lib/kubelet/pods/d26e69c6-ae7f-4fcd-895f-4dcd81062d67/volumes" Nov 02 18:05:57.183623 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.183576 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" podUID="23bfa30c-a0ac-4ec5-b12e-0d684c62ceff" containerName="main" containerID="cri-o://fbc0a8e64fd5342ec7b4f7f53020304bd3c689dbd2d646d4216007152a1e65f8" gracePeriod=45 Nov 02 18:05:57.329992 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.329930 4860 generic.go:356] "Generic (PLEG): container finished" podID="23bfa30c-a0ac-4ec5-b12e-0d684c62ceff" containerID="fbc0a8e64fd5342ec7b4f7f53020304bd3c689dbd2d646d4216007152a1e65f8" exitCode=0 Nov 02 18:05:57.330176 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.329999 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" event={"ID":"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff","Type":"ContainerDied","Data":"fbc0a8e64fd5342ec7b4f7f53020304bd3c689dbd2d646d4216007152a1e65f8"} Nov 02 18:05:57.551003 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.550669 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 18:05:57.702043 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.701963 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-kube-api-access-k9vmg\" (UniqueName: \"kubernetes.io/projected/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-kube-api-access-k9vmg\") pod \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " Nov 02 18:05:57.702200 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.702078 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-tmp-dir\") pod \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " Nov 02 18:05:57.702200 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.702108 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-tls-certs\" (UniqueName: \"kubernetes.io/secret/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-tls-certs\") pod \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " Nov 02 18:05:57.702200 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.702167 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-home\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-home\") pod \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " Nov 02 18:05:57.702364 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.702291 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-dshm\") pod \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " Nov 02 18:05:57.702426 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.702364 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-tmp-dir" pod "23bfa30c-a0ac-4ec5-b12e-0d684c62ceff" (UID: "23bfa30c-a0ac-4ec5-b12e-0d684c62ceff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:57.702525 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.702504 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-home" pod "23bfa30c-a0ac-4ec5-b12e-0d684c62ceff" (UID: "23bfa30c-a0ac-4ec5-b12e-0d684c62ceff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:57.702589 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.702559 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-model-cache\") pod \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\" (UID: \"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff\") " Nov 02 18:05:57.702811 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.702790 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-model-cache" pod "23bfa30c-a0ac-4ec5-b12e-0d684c62ceff" (UID: "23bfa30c-a0ac-4ec5-b12e-0d684c62ceff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:57.703668 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.703644 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:05:57.703668 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.703671 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:05:57.703804 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.703685 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:05:57.704632 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.704603 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-dshm" pod "23bfa30c-a0ac-4ec5-b12e-0d684c62ceff" (UID: "23bfa30c-a0ac-4ec5-b12e-0d684c62ceff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:05:57.704777 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.704761 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-tls-certs" pod "23bfa30c-a0ac-4ec5-b12e-0d684c62ceff" (UID: "23bfa30c-a0ac-4ec5-b12e-0d684c62ceff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:57.704888 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.704832 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-kube-api-access-k9vmg" pod "23bfa30c-a0ac-4ec5-b12e-0d684c62ceff" (UID: "23bfa30c-a0ac-4ec5-b12e-0d684c62ceff"). InnerVolumeSpecName "kube-api-access-k9vmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:57.805348 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.805306 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:05:57.805348 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.805342 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9vmg\" (UniqueName: \"kubernetes.io/projected/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-kube-api-access-k9vmg\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:05:57.805348 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:57.805353 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:05:58.341734 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:58.341692 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" Nov 02 18:05:58.342279 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:58.341699 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w" event={"ID":"23bfa30c-a0ac-4ec5-b12e-0d684c62ceff","Type":"ContainerDied","Data":"a23bd586349777c5091671301d82872bba50d7e0c07625009d144d0f5179f21c"} Nov 02 18:05:58.342279 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:58.341822 4860 scope.go:122] "RemoveContainer" containerID="fbc0a8e64fd5342ec7b4f7f53020304bd3c689dbd2d646d4216007152a1e65f8" Nov 02 18:05:58.407262 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:58.407231 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w"] Nov 02 18:05:58.412891 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:58.412861 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d5b905/test-model-d5b905-kserve-545db554fc-8mc9w"] Nov 02 18:05:59.503918 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:59.503883 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23bfa30c-a0ac-4ec5-b12e-0d684c62ceff" path="/var/lib/kubelet/pods/23bfa30c-a0ac-4ec5-b12e-0d684c62ceff/volumes" Nov 02 18:05:59.809532 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:05:59.801388 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx"] Nov 02 18:06:14.839930 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:14.839882 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" podUID="8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10" containerName="main" containerID="cri-o://492dec3242c3125df999443a8ec98a1bd9b84f229d8d436dc13c682e37d4c1a6" gracePeriod=45 Nov 02 18:06:15.220350 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.220317 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 18:06:15.334255 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.334212 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-tls-certs\") pod \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " Nov 02 18:06:15.334444 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.334287 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-dshm\") pod \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " Nov 02 18:06:15.334444 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.334345 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-home\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-home\") pod \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " Nov 02 18:06:15.334976 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.334688 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-model-cache\") pod \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " Nov 02 18:06:15.334976 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.334776 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-kube-api-access-fpjg2\" (UniqueName: \"kubernetes.io/projected/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-kube-api-access-fpjg2\") pod \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " Nov 02 18:06:15.334976 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.334809 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-tmp-dir\") pod \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\" (UID: \"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10\") " Nov 02 18:06:15.335288 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.335242 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-home" pod "8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10" (UID: "8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:06:15.335981 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.335618 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:06:15.335981 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.335876 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-tmp-dir" pod "8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10" (UID: "8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:06:15.335981 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.335923 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-model-cache" pod "8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10" (UID: "8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:06:15.341507 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.341465 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-dshm" pod "8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10" (UID: "8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:06:15.341812 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.341786 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-kube-api-access-fpjg2" pod "8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10" (UID: "8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10"). InnerVolumeSpecName "kube-api-access-fpjg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:06:15.342764 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.342725 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-tls-certs" pod "8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10" (UID: "8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:06:15.437836 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.437728 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpjg2\" (UniqueName: \"kubernetes.io/projected/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-kube-api-access-fpjg2\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:06:15.437836 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.437785 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:06:15.437836 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.437799 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:06:15.437836 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.437808 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:06:15.437836 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.437816 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:06:15.586090 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.586038 4860 generic.go:356] "Generic (PLEG): container finished" podID="8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10" containerID="492dec3242c3125df999443a8ec98a1bd9b84f229d8d436dc13c682e37d4c1a6" exitCode=0 Nov 02 18:06:15.586508 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.586101 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" Nov 02 18:06:15.586508 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.586263 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" event={"ID":"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10","Type":"ContainerDied","Data":"492dec3242c3125df999443a8ec98a1bd9b84f229d8d436dc13c682e37d4c1a6"} Nov 02 18:06:15.586508 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.586359 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx" event={"ID":"8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10","Type":"ContainerDied","Data":"9c5b5697ffe50cf7a9a0421e38e22cb8ad7a829fb63a97c11fdcceab85a379b2"} Nov 02 18:06:15.586508 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.586396 4860 scope.go:122] "RemoveContainer" containerID="492dec3242c3125df999443a8ec98a1bd9b84f229d8d436dc13c682e37d4c1a6" Nov 02 18:06:15.603911 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.603891 4860 scope.go:122] "RemoveContainer" containerID="492dec3242c3125df999443a8ec98a1bd9b84f229d8d436dc13c682e37d4c1a6" Nov 02 18:06:15.604604 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:06:15.604581 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492dec3242c3125df999443a8ec98a1bd9b84f229d8d436dc13c682e37d4c1a6\": container with ID starting with 492dec3242c3125df999443a8ec98a1bd9b84f229d8d436dc13c682e37d4c1a6 not found: ID does not exist" containerID="492dec3242c3125df999443a8ec98a1bd9b84f229d8d436dc13c682e37d4c1a6" Nov 02 18:06:15.604741 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.604611 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492dec3242c3125df999443a8ec98a1bd9b84f229d8d436dc13c682e37d4c1a6"} err="failed to get container status \"492dec3242c3125df999443a8ec98a1bd9b84f229d8d436dc13c682e37d4c1a6\": rpc error: code = NotFound desc = could not find container \"492dec3242c3125df999443a8ec98a1bd9b84f229d8d436dc13c682e37d4c1a6\": container with ID starting with 492dec3242c3125df999443a8ec98a1bd9b84f229d8d436dc13c682e37d4c1a6 not found: ID does not exist" Nov 02 18:06:15.650625 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.650526 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx"] Nov 02 18:06:15.655106 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:15.655071 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2c9b1d/test-model-2c9b1d-kserve-75f7cb9f89-9g7gx"] Nov 02 18:06:17.502229 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:06:17.502194 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10" path="/var/lib/kubelet/pods/8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10/volumes" Nov 02 18:06:21.794580 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:06:21.794517 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:06:51.808278 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:06:51.808216 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:07:21.772522 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:07:21.772485 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:07:51.773076 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:07:51.773034 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:08:14.514909 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.514814 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-55r9h"] Nov 02 18:08:14.515883 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.515843 4860 cpu_manager.go:414] "containerMap: removing container" podUID="48350106-29a0-4a22-a995-203997f1d909" containerName="main" Nov 02 18:08:14.515883 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.515883 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48350106-29a0-4a22-a995-203997f1d909" containerName="main" Nov 02 18:08:14.516012 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.515899 4860 cpu_manager.go:414] "containerMap: removing container" podUID="23bfa30c-a0ac-4ec5-b12e-0d684c62ceff" containerName="main" Nov 02 18:08:14.516012 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.515906 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23bfa30c-a0ac-4ec5-b12e-0d684c62ceff" containerName="main" Nov 02 18:08:14.516012 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.515921 4860 cpu_manager.go:414] "containerMap: removing container" podUID="d26e69c6-ae7f-4fcd-895f-4dcd81062d67" containerName="main" Nov 02 18:08:14.516012 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.515926 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26e69c6-ae7f-4fcd-895f-4dcd81062d67" containerName="main" Nov 02 18:08:14.516012 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.515933 4860 cpu_manager.go:414] "containerMap: removing container" podUID="8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10" containerName="main" Nov 02 18:08:14.516012 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.515939 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10" containerName="main" Nov 02 18:08:14.516253 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.516084 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="48350106-29a0-4a22-a995-203997f1d909" containerName="main" Nov 02 18:08:14.516253 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.516094 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="23bfa30c-a0ac-4ec5-b12e-0d684c62ceff" containerName="main" Nov 02 18:08:14.516253 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.516103 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26e69c6-ae7f-4fcd-895f-4dcd81062d67" containerName="main" Nov 02 18:08:14.516253 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.516110 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7782ec-d1ac-4fc7-9ce6-2b8866af3d10" containerName="main" Nov 02 18:08:14.521176 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.521151 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-55r9h" Nov 02 18:08:14.538305 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.538276 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-55r9h"] Nov 02 18:08:14.551071 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.549622 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vqf9q\"" Nov 02 18:08:14.561936 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.561341 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 18:08:14.561936 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.561428 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 18:08:14.624912 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.624822 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5z8c\" (UniqueName: \"kubernetes.io/projected/1ff6cd68-f65f-4a83-ae25-ae951b3b6f85-kube-api-access-g5z8c\") pod \"migrator-66df5d496b-55r9h\" (UID: \"1ff6cd68-f65f-4a83-ae25-ae951b3b6f85\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-55r9h" Nov 02 18:08:14.726807 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.726768 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5z8c\" (UniqueName: \"kubernetes.io/projected/1ff6cd68-f65f-4a83-ae25-ae951b3b6f85-kube-api-access-g5z8c\") pod \"migrator-66df5d496b-55r9h\" (UID: \"1ff6cd68-f65f-4a83-ae25-ae951b3b6f85\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-55r9h" Nov 02 18:08:14.750697 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.750665 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5z8c\" (UniqueName: \"kubernetes.io/projected/1ff6cd68-f65f-4a83-ae25-ae951b3b6f85-kube-api-access-g5z8c\") pod \"migrator-66df5d496b-55r9h\" (UID: \"1ff6cd68-f65f-4a83-ae25-ae951b3b6f85\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-55r9h" Nov 02 18:08:14.837757 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:14.837725 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-55r9h" Nov 02 18:08:15.150920 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:08:15.150842 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff6cd68_f65f_4a83_ae25_ae951b3b6f85.slice/crio-94bf6ddb285f979ccc11a7f5ab516f9f72bf84622774c57947dd82aaaa674052 WatchSource:0}: Error finding container 94bf6ddb285f979ccc11a7f5ab516f9f72bf84622774c57947dd82aaaa674052: Status 404 returned error can't find the container with id 94bf6ddb285f979ccc11a7f5ab516f9f72bf84622774c57947dd82aaaa674052 Nov 02 18:08:15.152490 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:15.152463 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-55r9h"] Nov 02 18:08:15.888081 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:15.888040 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-55r9h" event={"ID":"1ff6cd68-f65f-4a83-ae25-ae951b3b6f85","Type":"ContainerStarted","Data":"94bf6ddb285f979ccc11a7f5ab516f9f72bf84622774c57947dd82aaaa674052"} Nov 02 18:08:16.898922 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:16.898802 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-55r9h" event={"ID":"1ff6cd68-f65f-4a83-ae25-ae951b3b6f85","Type":"ContainerStarted","Data":"94d9c6fb34da55f28071a18a64e2265d062c209a148c46b3267e7482bc9d7090"} Nov 02 18:08:16.943247 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:08:16.943190 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-55r9h" podStartSLOduration=1.531482708 podStartE2EDuration="2.943172261s" totalImagesPullingTime="1.411689553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:16.939383322 +0000 UTC m=+2315.560743098" watchObservedRunningTime="2026-11-02 18:08:16.943172261 +0000 UTC m=+2315.564532036" Nov 02 18:08:21.803447 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:08:21.803399 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:08:51.772859 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:08:51.772809 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:09:21.773424 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:09:21.773382 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:09:42.051821 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:09:42.051791 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 18:09:42.063639 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:09:42.063608 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 18:09:42.065921 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:09:42.065900 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 18:09:42.077360 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:09:42.077341 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 18:09:51.778015 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:09:51.777971 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:10:10.278444 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:10.278351 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68"] Nov 02 18:10:10.279878 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:10.279443 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" podUID="39a03fdb-9060-4f2c-a96e-d78764568df8" containerName="istio-proxy" containerID="cri-o://ed501750397a99495279290d86a0add1ae10c39be1e6acacffde3bbf5022d625" gracePeriod=30 Nov 02 18:10:10.290940 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:10.290913 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh"] Nov 02 18:10:10.291355 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:10.291325 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" podUID="8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" containerName="istio-proxy" containerID="cri-o://31b4398bf86c26c964b4c307cb44eb619037686ca5b55915388dee728665d4d9" gracePeriod=30 Nov 02 18:10:10.294595 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:10.294572 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl"] Nov 02 18:10:10.294816 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:10.294773 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" podUID="151de229-e8e3-4bc3-8443-b12cfca9aab8" containerName="istio-proxy" containerID="cri-o://4c74414f378a26764acdd5f5aa77f5a7840a3d7eee8185b7077c602b64fdb46e" gracePeriod=30 Nov 02 18:10:10.737448 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:10.737136 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr"] Nov 02 18:10:10.737657 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:10.737393 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" podUID="76282517-6c00-4e83-b5f5-a6cfaa0dcad2" containerName="istio-proxy" containerID="cri-o://71da5e1a61cd83b0c52aaa1465cd9942dbe20a8750bf5364a1e8f96863c566ae" gracePeriod=30 Nov 02 18:10:10.745926 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:10.745879 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr"] Nov 02 18:10:10.746369 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:10.746140 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" podUID="5b585147-38f8-4a8f-be0b-d1ffaef9481d" containerName="istio-proxy" containerID="cri-o://0347bdf1fd1f77a0646b48b520e91b4a1a7cb0a9dcd686ffc1611e359031d770" gracePeriod=30 Nov 02 18:10:13.059372 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:13.059321 4860 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:10:13.059810 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:13.059404 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" podUID="76282517-6c00-4e83-b5f5-a6cfaa0dcad2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:10:14.677048 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:14.677015 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:10:14.677506 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:14.677073 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" podUID="39a03fdb-9060-4f2c-a96e-d78764568df8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:10:15.526937 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.526804 4860 generic.go:356] "Generic (PLEG): container finished" podID="39a03fdb-9060-4f2c-a96e-d78764568df8" containerID="ed501750397a99495279290d86a0add1ae10c39be1e6acacffde3bbf5022d625" exitCode=0 Nov 02 18:10:15.529041 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.529010 4860 generic.go:356] "Generic (PLEG): container finished" podID="8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" containerID="31b4398bf86c26c964b4c307cb44eb619037686ca5b55915388dee728665d4d9" exitCode=0 Nov 02 18:10:15.532898 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.531873 4860 generic.go:356] "Generic (PLEG): container finished" podID="151de229-e8e3-4bc3-8443-b12cfca9aab8" containerID="4c74414f378a26764acdd5f5aa77f5a7840a3d7eee8185b7077c602b64fdb46e" exitCode=0 Nov 02 18:10:15.533243 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.533217 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" event={"ID":"39a03fdb-9060-4f2c-a96e-d78764568df8","Type":"ContainerDied","Data":"ed501750397a99495279290d86a0add1ae10c39be1e6acacffde3bbf5022d625"} Nov 02 18:10:15.533336 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.533266 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" event={"ID":"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65","Type":"ContainerDied","Data":"31b4398bf86c26c964b4c307cb44eb619037686ca5b55915388dee728665d4d9"} Nov 02 18:10:15.533336 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.533292 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" event={"ID":"151de229-e8e3-4bc3-8443-b12cfca9aab8","Type":"ContainerDied","Data":"4c74414f378a26764acdd5f5aa77f5a7840a3d7eee8185b7077c602b64fdb46e"} Nov 02 18:10:15.538516 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.538486 4860 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 02 18:10:15.538647 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.538544 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" podUID="151de229-e8e3-4bc3-8443-b12cfca9aab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 02 18:10:15.632873 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.632512 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:10:15.712902 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.712277 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 18:10:15.716884 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.716816 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 18:10:15.744491 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.744454 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-credential-socket\") pod \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " Nov 02 18:10:15.744701 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.744516 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/151de229-e8e3-4bc3-8443-b12cfca9aab8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151de229-e8e3-4bc3-8443-b12cfca9aab8-istiod-ca-cert\") pod \"151de229-e8e3-4bc3-8443-b12cfca9aab8\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " Nov 02 18:10:15.744701 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.744554 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-token\" (UniqueName: \"kubernetes.io/projected/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-token\") pod \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " Nov 02 18:10:15.744701 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.744589 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-data\") pod \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " Nov 02 18:10:15.744701 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.744654 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-envoy\") pod \"151de229-e8e3-4bc3-8443-b12cfca9aab8\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " Nov 02 18:10:15.744701 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.744687 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-podinfo\") pod \"151de229-e8e3-4bc3-8443-b12cfca9aab8\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " Nov 02 18:10:15.745011 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.744742 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-token\" (UniqueName: \"kubernetes.io/projected/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-token\") pod \"39a03fdb-9060-4f2c-a96e-d78764568df8\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " Nov 02 18:10:15.745011 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.744786 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151de229-e8e3-4bc3-8443-b12cfca9aab8-kube-api-access-jdxq2\" (UniqueName: \"kubernetes.io/projected/151de229-e8e3-4bc3-8443-b12cfca9aab8-kube-api-access-jdxq2\") pod \"151de229-e8e3-4bc3-8443-b12cfca9aab8\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " Nov 02 18:10:15.745011 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.744786 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-credential-socket" pod "8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" (UID: "8b2fd7d0-d1d9-4087-9d88-25142f4a6b65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.745011 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.744814 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-podinfo\") pod \"39a03fdb-9060-4f2c-a96e-d78764568df8\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " Nov 02 18:10:15.745011 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.744894 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-credential-socket\") pod \"39a03fdb-9060-4f2c-a96e-d78764568df8\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " Nov 02 18:10:15.745011 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.744926 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-data\") pod \"39a03fdb-9060-4f2c-a96e-d78764568df8\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " Nov 02 18:10:15.745011 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.744982 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-token\" (UniqueName: \"kubernetes.io/projected/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-token\") pod \"151de229-e8e3-4bc3-8443-b12cfca9aab8\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " Nov 02 18:10:15.745472 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.745031 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-kube-api-access-tsdtt\" (UniqueName: \"kubernetes.io/projected/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-kube-api-access-tsdtt\") pod \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " Nov 02 18:10:15.745472 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.745069 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-workload-socket\") pod \"39a03fdb-9060-4f2c-a96e-d78764568df8\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " Nov 02 18:10:15.745472 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.745305 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istiod-ca-cert\") pod \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " Nov 02 18:10:15.745472 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.745372 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-data\") pod \"151de229-e8e3-4bc3-8443-b12cfca9aab8\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " Nov 02 18:10:15.745472 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.745412 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-workload-certs\") pod \"39a03fdb-9060-4f2c-a96e-d78764568df8\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " Nov 02 18:10:15.745472 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.745444 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a03fdb-9060-4f2c-a96e-d78764568df8-kube-api-access-grtgg\" (UniqueName: \"kubernetes.io/projected/39a03fdb-9060-4f2c-a96e-d78764568df8-kube-api-access-grtgg\") pod \"39a03fdb-9060-4f2c-a96e-d78764568df8\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " Nov 02 18:10:15.745782 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.745520 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-workload-socket\") pod \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " Nov 02 18:10:15.745782 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.745551 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-credential-socket\") pod \"151de229-e8e3-4bc3-8443-b12cfca9aab8\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " Nov 02 18:10:15.745782 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.745608 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-envoy\") pod \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " Nov 02 18:10:15.745782 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.745634 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-workload-certs\") pod \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " Nov 02 18:10:15.745782 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.745727 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-workload-socket\") pod \"151de229-e8e3-4bc3-8443-b12cfca9aab8\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " Nov 02 18:10:15.745782 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.745754 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39a03fdb-9060-4f2c-a96e-d78764568df8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a03fdb-9060-4f2c-a96e-d78764568df8-istiod-ca-cert\") pod \"39a03fdb-9060-4f2c-a96e-d78764568df8\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " Nov 02 18:10:15.746129 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.745808 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-workload-certs\") pod \"151de229-e8e3-4bc3-8443-b12cfca9aab8\" (UID: \"151de229-e8e3-4bc3-8443-b12cfca9aab8\") " Nov 02 18:10:15.746129 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.745911 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-podinfo\") pod \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\" (UID: \"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65\") " Nov 02 18:10:15.746129 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.745961 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-envoy\") pod \"39a03fdb-9060-4f2c-a96e-d78764568df8\" (UID: \"39a03fdb-9060-4f2c-a96e-d78764568df8\") " Nov 02 18:10:15.746129 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.746111 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-data" pod "8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" (UID: "8b2fd7d0-d1d9-4087-9d88-25142f4a6b65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.746348 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.746121 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istiod-ca-cert" pod "8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" (UID: "8b2fd7d0-d1d9-4087-9d88-25142f4a6b65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:15.746479 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.746450 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151de229-e8e3-4bc3-8443-b12cfca9aab8-istiod-ca-cert" pod "151de229-e8e3-4bc3-8443-b12cfca9aab8" (UID: "151de229-e8e3-4bc3-8443-b12cfca9aab8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:15.746803 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.746776 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-data" pod "39a03fdb-9060-4f2c-a96e-d78764568df8" (UID: "39a03fdb-9060-4f2c-a96e-d78764568df8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.747053 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.747030 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-data" pod "151de229-e8e3-4bc3-8443-b12cfca9aab8" (UID: "151de229-e8e3-4bc3-8443-b12cfca9aab8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.747193 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.747169 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.747272 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.747200 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.747272 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.747215 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.747272 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.747229 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.747272 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.747242 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151de229-e8e3-4bc3-8443-b12cfca9aab8-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.747732 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.747704 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-workload-socket" pod "39a03fdb-9060-4f2c-a96e-d78764568df8" (UID: "39a03fdb-9060-4f2c-a96e-d78764568df8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.747956 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.747931 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-workload-certs" pod "39a03fdb-9060-4f2c-a96e-d78764568df8" (UID: "39a03fdb-9060-4f2c-a96e-d78764568df8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.748105 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.748063 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-workload-certs" pod "8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" (UID: "8b2fd7d0-d1d9-4087-9d88-25142f4a6b65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.748183 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.748156 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-workload-socket" pod "151de229-e8e3-4bc3-8443-b12cfca9aab8" (UID: "151de229-e8e3-4bc3-8443-b12cfca9aab8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.748461 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.748440 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a03fdb-9060-4f2c-a96e-d78764568df8-istiod-ca-cert" pod "39a03fdb-9060-4f2c-a96e-d78764568df8" (UID: "39a03fdb-9060-4f2c-a96e-d78764568df8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:15.748682 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.748662 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-workload-certs" pod "151de229-e8e3-4bc3-8443-b12cfca9aab8" (UID: "151de229-e8e3-4bc3-8443-b12cfca9aab8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.750451 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.750312 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151de229-e8e3-4bc3-8443-b12cfca9aab8-kube-api-access-jdxq2" pod "151de229-e8e3-4bc3-8443-b12cfca9aab8" (UID: "151de229-e8e3-4bc3-8443-b12cfca9aab8"). InnerVolumeSpecName "kube-api-access-jdxq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:15.750645 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.750608 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-envoy" pod "151de229-e8e3-4bc3-8443-b12cfca9aab8" (UID: "151de229-e8e3-4bc3-8443-b12cfca9aab8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.751084 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.751055 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-credential-socket" pod "151de229-e8e3-4bc3-8443-b12cfca9aab8" (UID: "151de229-e8e3-4bc3-8443-b12cfca9aab8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.751290 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.751264 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-workload-socket" pod "8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" (UID: "8b2fd7d0-d1d9-4087-9d88-25142f4a6b65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.752018 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.751652 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-kube-api-access-tsdtt" pod "8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" (UID: "8b2fd7d0-d1d9-4087-9d88-25142f4a6b65"). InnerVolumeSpecName "kube-api-access-tsdtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:15.752018 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.751735 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-token" pod "39a03fdb-9060-4f2c-a96e-d78764568df8" (UID: "39a03fdb-9060-4f2c-a96e-d78764568df8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:15.752018 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.751791 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-token" pod "8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" (UID: "8b2fd7d0-d1d9-4087-9d88-25142f4a6b65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:15.752276 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.752023 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-token" pod "151de229-e8e3-4bc3-8443-b12cfca9aab8" (UID: "151de229-e8e3-4bc3-8443-b12cfca9aab8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:15.752464 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.752397 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-credential-socket" pod "39a03fdb-9060-4f2c-a96e-d78764568df8" (UID: "39a03fdb-9060-4f2c-a96e-d78764568df8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.752464 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.752446 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-podinfo" pod "8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" (UID: "8b2fd7d0-d1d9-4087-9d88-25142f4a6b65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:10:15.753508 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.753418 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-podinfo" pod "39a03fdb-9060-4f2c-a96e-d78764568df8" (UID: "39a03fdb-9060-4f2c-a96e-d78764568df8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:10:15.753615 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.753502 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a03fdb-9060-4f2c-a96e-d78764568df8-kube-api-access-grtgg" pod "39a03fdb-9060-4f2c-a96e-d78764568df8" (UID: "39a03fdb-9060-4f2c-a96e-d78764568df8"). InnerVolumeSpecName "kube-api-access-grtgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:15.753984 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.753962 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-podinfo" pod "151de229-e8e3-4bc3-8443-b12cfca9aab8" (UID: "151de229-e8e3-4bc3-8443-b12cfca9aab8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:10:15.754155 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.754136 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-envoy" pod "39a03fdb-9060-4f2c-a96e-d78764568df8" (UID: "39a03fdb-9060-4f2c-a96e-d78764568df8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.754775 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.754742 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-envoy" pod "8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" (UID: "8b2fd7d0-d1d9-4087-9d88-25142f4a6b65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:15.851424 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.850954 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.851424 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851257 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.851424 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851296 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.851424 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851312 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grtgg\" (UniqueName: \"kubernetes.io/projected/39a03fdb-9060-4f2c-a96e-d78764568df8-kube-api-access-grtgg\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.851424 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851329 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.851424 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851343 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.851424 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851357 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.851424 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851370 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.851424 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851383 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.851424 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851398 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a03fdb-9060-4f2c-a96e-d78764568df8-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.851424 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851414 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.851424 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851428 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.852211 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851441 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.852211 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851455 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.852211 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851469 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.852211 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851481 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.852211 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851495 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.852211 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851510 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdxq2\" (UniqueName: \"kubernetes.io/projected/151de229-e8e3-4bc3-8443-b12cfca9aab8-kube-api-access-jdxq2\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.852211 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851524 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a03fdb-9060-4f2c-a96e-d78764568df8-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.852211 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851538 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a03fdb-9060-4f2c-a96e-d78764568df8-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.852211 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851550 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151de229-e8e3-4bc3-8443-b12cfca9aab8-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.852211 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.851563 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsdtt\" (UniqueName: \"kubernetes.io/projected/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65-kube-api-access-tsdtt\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:15.928506 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.928469 4860 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 18:10:15.928715 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:15.928543 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" podUID="5b585147-38f8-4a8f-be0b-d1ffaef9481d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 18:10:16.051169 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.051138 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:10:16.082578 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.082553 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:10:16.178752 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.175631 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-envoy\") pod \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " Nov 02 18:10:16.178752 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.175718 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-credential-socket\") pod \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " Nov 02 18:10:16.178752 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.175763 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-token\" (UniqueName: \"kubernetes.io/projected/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-token\") pod \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " Nov 02 18:10:16.178752 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.175791 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-data\") pod \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " Nov 02 18:10:16.178752 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.175832 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-workload-certs\") pod \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " Nov 02 18:10:16.178752 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.175900 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-credential-socket\") pod \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " Nov 02 18:10:16.178752 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.175928 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-envoy\") pod \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " Nov 02 18:10:16.178752 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.176036 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-data\") pod \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " Nov 02 18:10:16.178752 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.176073 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-podinfo\") pod \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " Nov 02 18:10:16.178752 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.176135 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-token\" (UniqueName: \"kubernetes.io/projected/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-token\") pod \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " Nov 02 18:10:16.178752 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.176170 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-workload-certs\") pod \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " Nov 02 18:10:16.178752 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.176211 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istiod-ca-cert\") pod \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " Nov 02 18:10:16.178752 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.176305 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-kube-api-access-4zccp\" (UniqueName: \"kubernetes.io/projected/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-kube-api-access-4zccp\") pod \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " Nov 02 18:10:16.178752 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.176428 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-workload-socket\") pod \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " Nov 02 18:10:16.178752 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.176467 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-workload-socket\") pod \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " Nov 02 18:10:16.178752 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.176553 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-podinfo\") pod \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " Nov 02 18:10:16.178752 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.176747 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istiod-ca-cert\") pod \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\" (UID: \"76282517-6c00-4e83-b5f5-a6cfaa0dcad2\") " Nov 02 18:10:16.178752 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.177014 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b585147-38f8-4a8f-be0b-d1ffaef9481d-kube-api-access-hbspf\" (UniqueName: \"kubernetes.io/projected/5b585147-38f8-4a8f-be0b-d1ffaef9481d-kube-api-access-hbspf\") pod \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\" (UID: \"5b585147-38f8-4a8f-be0b-d1ffaef9481d\") " Nov 02 18:10:16.181118 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.180882 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b585147-38f8-4a8f-be0b-d1ffaef9481d-kube-api-access-hbspf" pod "5b585147-38f8-4a8f-be0b-d1ffaef9481d" (UID: "5b585147-38f8-4a8f-be0b-d1ffaef9481d"). InnerVolumeSpecName "kube-api-access-hbspf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:16.182745 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.182152 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-credential-socket" pod "76282517-6c00-4e83-b5f5-a6cfaa0dcad2" (UID: "76282517-6c00-4e83-b5f5-a6cfaa0dcad2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:16.185721 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.185527 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-envoy" pod "5b585147-38f8-4a8f-be0b-d1ffaef9481d" (UID: "5b585147-38f8-4a8f-be0b-d1ffaef9481d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:16.186077 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.185900 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-workload-socket" pod "76282517-6c00-4e83-b5f5-a6cfaa0dcad2" (UID: "76282517-6c00-4e83-b5f5-a6cfaa0dcad2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:16.186378 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.186207 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-credential-socket" pod "5b585147-38f8-4a8f-be0b-d1ffaef9481d" (UID: "5b585147-38f8-4a8f-be0b-d1ffaef9481d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:16.186667 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.186504 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-data" pod "76282517-6c00-4e83-b5f5-a6cfaa0dcad2" (UID: "76282517-6c00-4e83-b5f5-a6cfaa0dcad2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:16.186977 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.186794 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-workload-certs" pod "5b585147-38f8-4a8f-be0b-d1ffaef9481d" (UID: "5b585147-38f8-4a8f-be0b-d1ffaef9481d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:16.187374 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.187118 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-data" pod "5b585147-38f8-4a8f-be0b-d1ffaef9481d" (UID: "5b585147-38f8-4a8f-be0b-d1ffaef9481d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:16.201098 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.199562 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-envoy" pod "76282517-6c00-4e83-b5f5-a6cfaa0dcad2" (UID: "76282517-6c00-4e83-b5f5-a6cfaa0dcad2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:16.201098 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.200804 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-workload-certs" pod "76282517-6c00-4e83-b5f5-a6cfaa0dcad2" (UID: "76282517-6c00-4e83-b5f5-a6cfaa0dcad2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:16.201901 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.201699 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-workload-socket" pod "5b585147-38f8-4a8f-be0b-d1ffaef9481d" (UID: "5b585147-38f8-4a8f-be0b-d1ffaef9481d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:10:16.202015 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.201927 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-token" pod "76282517-6c00-4e83-b5f5-a6cfaa0dcad2" (UID: "76282517-6c00-4e83-b5f5-a6cfaa0dcad2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:16.203495 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.202228 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istiod-ca-cert" pod "76282517-6c00-4e83-b5f5-a6cfaa0dcad2" (UID: "76282517-6c00-4e83-b5f5-a6cfaa0dcad2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:16.203495 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.202790 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istiod-ca-cert" pod "5b585147-38f8-4a8f-be0b-d1ffaef9481d" (UID: "5b585147-38f8-4a8f-be0b-d1ffaef9481d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:16.204399 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.204360 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-kube-api-access-4zccp" pod "76282517-6c00-4e83-b5f5-a6cfaa0dcad2" (UID: "76282517-6c00-4e83-b5f5-a6cfaa0dcad2"). InnerVolumeSpecName "kube-api-access-4zccp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:16.207721 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.207688 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-token" pod "5b585147-38f8-4a8f-be0b-d1ffaef9481d" (UID: "5b585147-38f8-4a8f-be0b-d1ffaef9481d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:16.208162 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.208137 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-podinfo" pod "5b585147-38f8-4a8f-be0b-d1ffaef9481d" (UID: "5b585147-38f8-4a8f-be0b-d1ffaef9481d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:10:16.208966 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.208941 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-podinfo" pod "76282517-6c00-4e83-b5f5-a6cfaa0dcad2" (UID: "76282517-6c00-4e83-b5f5-a6cfaa0dcad2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:10:16.279917 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.279881 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.280111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.279931 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbspf\" (UniqueName: \"kubernetes.io/projected/5b585147-38f8-4a8f-be0b-d1ffaef9481d-kube-api-access-hbspf\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.280111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.279942 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.280111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.279951 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.280111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.279960 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.280111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.279968 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.280111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.279976 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.280111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.279984 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.280111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.279992 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.280111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.280000 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.280111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.280008 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.280111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.280015 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.280111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.280023 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.280111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.280031 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b585147-38f8-4a8f-be0b-d1ffaef9481d-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.280111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.280038 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zccp\" (UniqueName: \"kubernetes.io/projected/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-kube-api-access-4zccp\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.280111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.280046 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.280111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.280054 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b585147-38f8-4a8f-be0b-d1ffaef9481d-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.280111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.280062 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76282517-6c00-4e83-b5f5-a6cfaa0dcad2-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:16.542132 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.542042 4860 generic.go:356] "Generic (PLEG): container finished" podID="76282517-6c00-4e83-b5f5-a6cfaa0dcad2" containerID="71da5e1a61cd83b0c52aaa1465cd9942dbe20a8750bf5364a1e8f96863c566ae" exitCode=0 Nov 02 18:10:16.542132 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.542104 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" Nov 02 18:10:16.542132 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.542121 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" event={"ID":"76282517-6c00-4e83-b5f5-a6cfaa0dcad2","Type":"ContainerDied","Data":"71da5e1a61cd83b0c52aaa1465cd9942dbe20a8750bf5364a1e8f96863c566ae"} Nov 02 18:10:16.542373 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.542180 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr" event={"ID":"76282517-6c00-4e83-b5f5-a6cfaa0dcad2","Type":"ContainerDied","Data":"7622afd83d6bcd6424e3e40d18ef54c095dbdd326d1cb6bb0b338cb614fabf11"} Nov 02 18:10:16.542373 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.542213 4860 scope.go:122] "RemoveContainer" containerID="71da5e1a61cd83b0c52aaa1465cd9942dbe20a8750bf5364a1e8f96863c566ae" Nov 02 18:10:16.544102 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.544082 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" Nov 02 18:10:16.544237 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.544120 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl" event={"ID":"151de229-e8e3-4bc3-8443-b12cfca9aab8","Type":"ContainerDied","Data":"76ca239226d5459b9df1da4c4a914e4cd89a7bac3f6705ced7be79861f0616b1"} Nov 02 18:10:16.545872 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.545830 4860 generic.go:356] "Generic (PLEG): container finished" podID="5b585147-38f8-4a8f-be0b-d1ffaef9481d" containerID="0347bdf1fd1f77a0646b48b520e91b4a1a7cb0a9dcd686ffc1611e359031d770" exitCode=0 Nov 02 18:10:16.545986 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.545879 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" event={"ID":"5b585147-38f8-4a8f-be0b-d1ffaef9481d","Type":"ContainerDied","Data":"0347bdf1fd1f77a0646b48b520e91b4a1a7cb0a9dcd686ffc1611e359031d770"} Nov 02 18:10:16.545986 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.545916 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" Nov 02 18:10:16.545986 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.545947 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr" event={"ID":"5b585147-38f8-4a8f-be0b-d1ffaef9481d","Type":"ContainerDied","Data":"ad1c66cc7249e727abaa2052b649ebd5620a279e49f13ee68e848b988e72c2b6"} Nov 02 18:10:16.547681 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.547659 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" Nov 02 18:10:16.547681 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.547671 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68" event={"ID":"39a03fdb-9060-4f2c-a96e-d78764568df8","Type":"ContainerDied","Data":"c30ea041248743c96328ffe2871eb9eb8562c58803e0929c32da6cb7fa6a1c15"} Nov 02 18:10:16.552997 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.552900 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" event={"ID":"8b2fd7d0-d1d9-4087-9d88-25142f4a6b65","Type":"ContainerDied","Data":"5fe51f59c725ddd1dcfdcebea620964e7a4aa0bc092a34615314449fd34f7350"} Nov 02 18:10:16.552997 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.552929 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh" Nov 02 18:10:16.559722 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.559612 4860 scope.go:122] "RemoveContainer" containerID="71da5e1a61cd83b0c52aaa1465cd9942dbe20a8750bf5364a1e8f96863c566ae" Nov 02 18:10:16.560121 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:10:16.560086 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71da5e1a61cd83b0c52aaa1465cd9942dbe20a8750bf5364a1e8f96863c566ae\": container with ID starting with 71da5e1a61cd83b0c52aaa1465cd9942dbe20a8750bf5364a1e8f96863c566ae not found: ID does not exist" containerID="71da5e1a61cd83b0c52aaa1465cd9942dbe20a8750bf5364a1e8f96863c566ae" Nov 02 18:10:16.560237 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.560123 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71da5e1a61cd83b0c52aaa1465cd9942dbe20a8750bf5364a1e8f96863c566ae"} err="failed to get container status \"71da5e1a61cd83b0c52aaa1465cd9942dbe20a8750bf5364a1e8f96863c566ae\": rpc error: code = NotFound desc = could not find container \"71da5e1a61cd83b0c52aaa1465cd9942dbe20a8750bf5364a1e8f96863c566ae\": container with ID starting with 71da5e1a61cd83b0c52aaa1465cd9942dbe20a8750bf5364a1e8f96863c566ae not found: ID does not exist" Nov 02 18:10:16.560237 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.560147 4860 scope.go:122] "RemoveContainer" containerID="4c74414f378a26764acdd5f5aa77f5a7840a3d7eee8185b7077c602b64fdb46e" Nov 02 18:10:16.573065 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.573042 4860 scope.go:122] "RemoveContainer" containerID="0347bdf1fd1f77a0646b48b520e91b4a1a7cb0a9dcd686ffc1611e359031d770" Nov 02 18:10:16.586719 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.586697 4860 scope.go:122] "RemoveContainer" containerID="0347bdf1fd1f77a0646b48b520e91b4a1a7cb0a9dcd686ffc1611e359031d770" Nov 02 18:10:16.587087 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:10:16.587062 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0347bdf1fd1f77a0646b48b520e91b4a1a7cb0a9dcd686ffc1611e359031d770\": container with ID starting with 0347bdf1fd1f77a0646b48b520e91b4a1a7cb0a9dcd686ffc1611e359031d770 not found: ID does not exist" containerID="0347bdf1fd1f77a0646b48b520e91b4a1a7cb0a9dcd686ffc1611e359031d770" Nov 02 18:10:16.587207 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.587098 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0347bdf1fd1f77a0646b48b520e91b4a1a7cb0a9dcd686ffc1611e359031d770"} err="failed to get container status \"0347bdf1fd1f77a0646b48b520e91b4a1a7cb0a9dcd686ffc1611e359031d770\": rpc error: code = NotFound desc = could not find container \"0347bdf1fd1f77a0646b48b520e91b4a1a7cb0a9dcd686ffc1611e359031d770\": container with ID starting with 0347bdf1fd1f77a0646b48b520e91b4a1a7cb0a9dcd686ffc1611e359031d770 not found: ID does not exist" Nov 02 18:10:16.587207 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.587122 4860 scope.go:122] "RemoveContainer" containerID="ed501750397a99495279290d86a0add1ae10c39be1e6acacffde3bbf5022d625" Nov 02 18:10:16.598306 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.598288 4860 scope.go:122] "RemoveContainer" containerID="31b4398bf86c26c964b4c307cb44eb619037686ca5b55915388dee728665d4d9" Nov 02 18:10:16.631177 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.631147 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr"] Nov 02 18:10:16.642809 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.642781 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jdhvr"] Nov 02 18:10:16.703179 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.703147 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh"] Nov 02 18:10:16.708359 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.708328 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-m22jh"] Nov 02 18:10:16.743884 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.743814 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr"] Nov 02 18:10:16.748882 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.748826 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-jrbkr"] Nov 02 18:10:16.782729 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.782696 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl"] Nov 02 18:10:16.788157 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.788126 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-zrcwl"] Nov 02 18:10:16.821049 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.821023 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68"] Nov 02 18:10:16.827059 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:16.827033 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-mhl68"] Nov 02 18:10:17.503549 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:17.503520 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151de229-e8e3-4bc3-8443-b12cfca9aab8" path="/var/lib/kubelet/pods/151de229-e8e3-4bc3-8443-b12cfca9aab8/volumes" Nov 02 18:10:17.504153 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:17.504134 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39a03fdb-9060-4f2c-a96e-d78764568df8" path="/var/lib/kubelet/pods/39a03fdb-9060-4f2c-a96e-d78764568df8/volumes" Nov 02 18:10:17.504680 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:17.504666 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b585147-38f8-4a8f-be0b-d1ffaef9481d" path="/var/lib/kubelet/pods/5b585147-38f8-4a8f-be0b-d1ffaef9481d/volumes" Nov 02 18:10:17.505528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:17.505512 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76282517-6c00-4e83-b5f5-a6cfaa0dcad2" path="/var/lib/kubelet/pods/76282517-6c00-4e83-b5f5-a6cfaa0dcad2/volumes" Nov 02 18:10:17.506021 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:17.506000 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" path="/var/lib/kubelet/pods/8b2fd7d0-d1d9-4087-9d88-25142f4a6b65/volumes" Nov 02 18:10:21.806034 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:10:21.805991 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:10:30.434050 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:30.434015 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg"] Nov 02 18:10:30.434464 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:30.434273 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" podUID="16e30f93-2270-4fb7-bfc2-3e48edf5ecc1" containerName="maas-api" containerID="cri-o://237a5fa79b8e8ff9f658274b472e3363be975f4d890f0a9f0cdce21fad61b343" gracePeriod=30 Nov 02 18:10:31.036929 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:31.036258 4860 generic.go:356] "Generic (PLEG): container finished" podID="16e30f93-2270-4fb7-bfc2-3e48edf5ecc1" containerID="237a5fa79b8e8ff9f658274b472e3363be975f4d890f0a9f0cdce21fad61b343" exitCode=0 Nov 02 18:10:31.036929 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:31.036392 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" event={"ID":"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1","Type":"ContainerDied","Data":"237a5fa79b8e8ff9f658274b472e3363be975f4d890f0a9f0cdce21fad61b343"} Nov 02 18:10:31.390178 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:31.390142 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" Nov 02 18:10:31.503832 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:31.503797 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-metrics-certs\") pod \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\" (UID: \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\") " Nov 02 18:10:31.503832 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:31.503902 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-openshift-service-ca\") pod \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\" (UID: \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\") " Nov 02 18:10:31.504443 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:31.504054 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-kube-api-access-vpl5r\" (UniqueName: \"kubernetes.io/projected/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-kube-api-access-vpl5r\") pod \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\" (UID: \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\") " Nov 02 18:10:31.504443 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:31.504341 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-openshift-service-ca" pod "16e30f93-2270-4fb7-bfc2-3e48edf5ecc1" (UID: "16e30f93-2270-4fb7-bfc2-3e48edf5ecc1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:31.504546 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:31.504454 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-maas-api-tls\") pod \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\" (UID: \"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1\") " Nov 02 18:10:31.505545 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:31.505515 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:31.506469 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:31.506441 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-metrics-certs" pod "16e30f93-2270-4fb7-bfc2-3e48edf5ecc1" (UID: "16e30f93-2270-4fb7-bfc2-3e48edf5ecc1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:10:31.506601 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:31.506489 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-kube-api-access-vpl5r" pod "16e30f93-2270-4fb7-bfc2-3e48edf5ecc1" (UID: "16e30f93-2270-4fb7-bfc2-3e48edf5ecc1"). InnerVolumeSpecName "kube-api-access-vpl5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:31.507005 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:31.506981 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-maas-api-tls" pod "16e30f93-2270-4fb7-bfc2-3e48edf5ecc1" (UID: "16e30f93-2270-4fb7-bfc2-3e48edf5ecc1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:10:31.607428 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:31.607335 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:31.607428 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:31.607371 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpl5r\" (UniqueName: \"kubernetes.io/projected/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-kube-api-access-vpl5r\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:31.607428 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:31.607387 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:32.047077 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:32.046991 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" Nov 02 18:10:32.047077 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:32.047031 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg" event={"ID":"16e30f93-2270-4fb7-bfc2-3e48edf5ecc1","Type":"ContainerDied","Data":"16fc2400c0eed3c4d77a869b1785ba808df0f6f670d0975f3b318d969afaa689"} Nov 02 18:10:32.047297 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:32.047093 4860 scope.go:122] "RemoveContainer" containerID="237a5fa79b8e8ff9f658274b472e3363be975f4d890f0a9f0cdce21fad61b343" Nov 02 18:10:32.102305 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:32.102269 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg"] Nov 02 18:10:32.107599 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:32.107562 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fbbbf8-8498cd66b9-286kg"] Nov 02 18:10:33.225702 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:33.225651 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq"] Nov 02 18:10:33.226218 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:33.225929 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" podUID="d7e0682c-4519-452a-ab21-9781064a6260" containerName="maas-api" containerID="cri-o://45a94738647f7ba57a93a9ee4b6458dfacf4ec1ef9c72e9e3a5677d0fb802344" gracePeriod=30 Nov 02 18:10:33.716313 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:33.714116 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e30f93-2270-4fb7-bfc2-3e48edf5ecc1" path="/var/lib/kubelet/pods/16e30f93-2270-4fb7-bfc2-3e48edf5ecc1/volumes" Nov 02 18:10:33.747204 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:33.747170 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" Nov 02 18:10:33.921875 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:33.920521 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7e0682c-4519-452a-ab21-9781064a6260-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7e0682c-4519-452a-ab21-9781064a6260-maas-api-tls\") pod \"d7e0682c-4519-452a-ab21-9781064a6260\" (UID: \"d7e0682c-4519-452a-ab21-9781064a6260\") " Nov 02 18:10:33.921875 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:33.920603 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e0682c-4519-452a-ab21-9781064a6260-kube-api-access-zzk6t\" (UniqueName: \"kubernetes.io/projected/d7e0682c-4519-452a-ab21-9781064a6260-kube-api-access-zzk6t\") pod \"d7e0682c-4519-452a-ab21-9781064a6260\" (UID: \"d7e0682c-4519-452a-ab21-9781064a6260\") " Nov 02 18:10:33.921875 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:33.920699 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7e0682c-4519-452a-ab21-9781064a6260-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e0682c-4519-452a-ab21-9781064a6260-openshift-service-ca\") pod \"d7e0682c-4519-452a-ab21-9781064a6260\" (UID: \"d7e0682c-4519-452a-ab21-9781064a6260\") " Nov 02 18:10:33.921875 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:33.920867 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7e0682c-4519-452a-ab21-9781064a6260-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e0682c-4519-452a-ab21-9781064a6260-metrics-certs\") pod \"d7e0682c-4519-452a-ab21-9781064a6260\" (UID: \"d7e0682c-4519-452a-ab21-9781064a6260\") " Nov 02 18:10:33.929489 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:33.929188 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e0682c-4519-452a-ab21-9781064a6260-openshift-service-ca" pod "d7e0682c-4519-452a-ab21-9781064a6260" (UID: "d7e0682c-4519-452a-ab21-9781064a6260"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:10:33.971933 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:33.944236 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7e0682c-4519-452a-ab21-9781064a6260-metrics-certs" pod "d7e0682c-4519-452a-ab21-9781064a6260" (UID: "d7e0682c-4519-452a-ab21-9781064a6260"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:10:33.996871 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:33.996799 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e0682c-4519-452a-ab21-9781064a6260-kube-api-access-zzk6t" pod "d7e0682c-4519-452a-ab21-9781064a6260" (UID: "d7e0682c-4519-452a-ab21-9781064a6260"). InnerVolumeSpecName "kube-api-access-zzk6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:10:33.998049 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:33.998002 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7e0682c-4519-452a-ab21-9781064a6260-maas-api-tls" pod "d7e0682c-4519-452a-ab21-9781064a6260" (UID: "d7e0682c-4519-452a-ab21-9781064a6260"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:10:34.027838 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:34.026533 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e0682c-4519-452a-ab21-9781064a6260-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:34.027838 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:34.026570 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7e0682c-4519-452a-ab21-9781064a6260-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:34.027838 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:34.026580 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzk6t\" (UniqueName: \"kubernetes.io/projected/d7e0682c-4519-452a-ab21-9781064a6260-kube-api-access-zzk6t\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:34.027838 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:34.026592 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e0682c-4519-452a-ab21-9781064a6260-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:10:34.112889 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:34.112833 4860 generic.go:356] "Generic (PLEG): container finished" podID="d7e0682c-4519-452a-ab21-9781064a6260" containerID="45a94738647f7ba57a93a9ee4b6458dfacf4ec1ef9c72e9e3a5677d0fb802344" exitCode=0 Nov 02 18:10:34.113073 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:34.112915 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" event={"ID":"d7e0682c-4519-452a-ab21-9781064a6260","Type":"ContainerDied","Data":"45a94738647f7ba57a93a9ee4b6458dfacf4ec1ef9c72e9e3a5677d0fb802344"} Nov 02 18:10:34.113073 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:34.112946 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" Nov 02 18:10:34.113073 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:34.113013 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq" event={"ID":"d7e0682c-4519-452a-ab21-9781064a6260","Type":"ContainerDied","Data":"4de9456a47e4b2fa65256a7c9f3c78d467b01a3e8aba4b28bd1512c036c68eaa"} Nov 02 18:10:34.113073 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:34.113054 4860 scope.go:122] "RemoveContainer" containerID="45a94738647f7ba57a93a9ee4b6458dfacf4ec1ef9c72e9e3a5677d0fb802344" Nov 02 18:10:34.126972 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:34.126947 4860 scope.go:122] "RemoveContainer" containerID="45a94738647f7ba57a93a9ee4b6458dfacf4ec1ef9c72e9e3a5677d0fb802344" Nov 02 18:10:34.127334 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:10:34.127303 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a94738647f7ba57a93a9ee4b6458dfacf4ec1ef9c72e9e3a5677d0fb802344\": container with ID starting with 45a94738647f7ba57a93a9ee4b6458dfacf4ec1ef9c72e9e3a5677d0fb802344 not found: ID does not exist" containerID="45a94738647f7ba57a93a9ee4b6458dfacf4ec1ef9c72e9e3a5677d0fb802344" Nov 02 18:10:34.127415 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:34.127346 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a94738647f7ba57a93a9ee4b6458dfacf4ec1ef9c72e9e3a5677d0fb802344"} err="failed to get container status \"45a94738647f7ba57a93a9ee4b6458dfacf4ec1ef9c72e9e3a5677d0fb802344\": rpc error: code = NotFound desc = could not find container \"45a94738647f7ba57a93a9ee4b6458dfacf4ec1ef9c72e9e3a5677d0fb802344\": container with ID starting with 45a94738647f7ba57a93a9ee4b6458dfacf4ec1ef9c72e9e3a5677d0fb802344 not found: ID does not exist" Nov 02 18:10:34.190819 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:34.190777 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq"] Nov 02 18:10:34.196375 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:34.196344 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b2fc9f-6b69b4bc67-c9xhq"] Nov 02 18:10:35.501333 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:35.501297 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e0682c-4519-452a-ab21-9781064a6260" path="/var/lib/kubelet/pods/d7e0682c-4519-452a-ab21-9781064a6260/volumes" Nov 02 18:10:51.772937 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:10:51.772897 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:10:59.276307 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:59.276219 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd"] Nov 02 18:10:59.276710 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:59.276489 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" podUID="ceba04e6-9968-4301-aaa3-6372edea5720" containerName="maas-api" containerID="cri-o://337bf193f9d841beae50b0763ffc4658f22c84c3df103a4610fe9573c8eed447" gracePeriod=30 Nov 02 18:10:59.952887 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:59.951132 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" Nov 02 18:11:00.001875 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:59.996600 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceba04e6-9968-4301-aaa3-6372edea5720-kube-api-access-zprkb\" (UniqueName: \"kubernetes.io/projected/ceba04e6-9968-4301-aaa3-6372edea5720-kube-api-access-zprkb\") pod \"ceba04e6-9968-4301-aaa3-6372edea5720\" (UID: \"ceba04e6-9968-4301-aaa3-6372edea5720\") " Nov 02 18:11:00.001875 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:59.996672 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceba04e6-9968-4301-aaa3-6372edea5720-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceba04e6-9968-4301-aaa3-6372edea5720-maas-api-tls\") pod \"ceba04e6-9968-4301-aaa3-6372edea5720\" (UID: \"ceba04e6-9968-4301-aaa3-6372edea5720\") " Nov 02 18:11:00.001875 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:59.996711 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceba04e6-9968-4301-aaa3-6372edea5720-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceba04e6-9968-4301-aaa3-6372edea5720-openshift-service-ca\") pod \"ceba04e6-9968-4301-aaa3-6372edea5720\" (UID: \"ceba04e6-9968-4301-aaa3-6372edea5720\") " Nov 02 18:11:00.001875 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:10:59.996754 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceba04e6-9968-4301-aaa3-6372edea5720-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceba04e6-9968-4301-aaa3-6372edea5720-metrics-certs\") pod \"ceba04e6-9968-4301-aaa3-6372edea5720\" (UID: \"ceba04e6-9968-4301-aaa3-6372edea5720\") " Nov 02 18:11:00.006875 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:00.005267 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceba04e6-9968-4301-aaa3-6372edea5720-openshift-service-ca" pod "ceba04e6-9968-4301-aaa3-6372edea5720" (UID: "ceba04e6-9968-4301-aaa3-6372edea5720"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:00.033123 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:00.031376 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceba04e6-9968-4301-aaa3-6372edea5720-kube-api-access-zprkb" pod "ceba04e6-9968-4301-aaa3-6372edea5720" (UID: "ceba04e6-9968-4301-aaa3-6372edea5720"). InnerVolumeSpecName "kube-api-access-zprkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:00.063062 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:00.060370 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceba04e6-9968-4301-aaa3-6372edea5720-metrics-certs" pod "ceba04e6-9968-4301-aaa3-6372edea5720" (UID: "ceba04e6-9968-4301-aaa3-6372edea5720"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:11:00.069729 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:00.064595 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceba04e6-9968-4301-aaa3-6372edea5720-maas-api-tls" pod "ceba04e6-9968-4301-aaa3-6372edea5720" (UID: "ceba04e6-9968-4301-aaa3-6372edea5720"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:11:00.100200 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:00.100155 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zprkb\" (UniqueName: \"kubernetes.io/projected/ceba04e6-9968-4301-aaa3-6372edea5720-kube-api-access-zprkb\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:00.100200 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:00.100192 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceba04e6-9968-4301-aaa3-6372edea5720-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:00.100463 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:00.100208 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceba04e6-9968-4301-aaa3-6372edea5720-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:00.100463 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:00.100222 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceba04e6-9968-4301-aaa3-6372edea5720-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:00.403138 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:00.403023 4860 generic.go:356] "Generic (PLEG): container finished" podID="ceba04e6-9968-4301-aaa3-6372edea5720" containerID="337bf193f9d841beae50b0763ffc4658f22c84c3df103a4610fe9573c8eed447" exitCode=0 Nov 02 18:11:00.403710 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:00.403299 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" event={"ID":"ceba04e6-9968-4301-aaa3-6372edea5720","Type":"ContainerDied","Data":"337bf193f9d841beae50b0763ffc4658f22c84c3df103a4610fe9573c8eed447"} Nov 02 18:11:00.403710 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:00.403358 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" event={"ID":"ceba04e6-9968-4301-aaa3-6372edea5720","Type":"ContainerDied","Data":"256a0131e18b82d9b1c7ed932741bcbb48de501c2f67ddd246ccbb7597f5b0df"} Nov 02 18:11:00.405189 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:00.405155 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd" Nov 02 18:11:00.405328 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:00.405184 4860 scope.go:122] "RemoveContainer" containerID="337bf193f9d841beae50b0763ffc4658f22c84c3df103a4610fe9573c8eed447" Nov 02 18:11:00.422297 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:00.422273 4860 scope.go:122] "RemoveContainer" containerID="337bf193f9d841beae50b0763ffc4658f22c84c3df103a4610fe9573c8eed447" Nov 02 18:11:00.422627 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:11:00.422604 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"337bf193f9d841beae50b0763ffc4658f22c84c3df103a4610fe9573c8eed447\": container with ID starting with 337bf193f9d841beae50b0763ffc4658f22c84c3df103a4610fe9573c8eed447 not found: ID does not exist" containerID="337bf193f9d841beae50b0763ffc4658f22c84c3df103a4610fe9573c8eed447" Nov 02 18:11:00.422710 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:00.422637 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337bf193f9d841beae50b0763ffc4658f22c84c3df103a4610fe9573c8eed447"} err="failed to get container status \"337bf193f9d841beae50b0763ffc4658f22c84c3df103a4610fe9573c8eed447\": rpc error: code = NotFound desc = could not find container \"337bf193f9d841beae50b0763ffc4658f22c84c3df103a4610fe9573c8eed447\": container with ID starting with 337bf193f9d841beae50b0763ffc4658f22c84c3df103a4610fe9573c8eed447 not found: ID does not exist" Nov 02 18:11:00.461415 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:00.461384 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd"] Nov 02 18:11:00.473408 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:00.473378 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5fd3e0-ffccc4c84-g2vdd"] Nov 02 18:11:01.502437 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:01.502407 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceba04e6-9968-4301-aaa3-6372edea5720" path="/var/lib/kubelet/pods/ceba04e6-9968-4301-aaa3-6372edea5720/volumes" Nov 02 18:11:01.922980 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:01.922942 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb"] Nov 02 18:11:01.923294 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:01.923200 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" podUID="6277b932-8a69-4947-afe2-0946a7c6f177" containerName="maas-api" containerID="cri-o://fec8d6a1326506e9d2303346e455516e0ad98a3b27c19b55905f9a6c2f591199" gracePeriod=30 Nov 02 18:11:02.500874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:02.494536 4860 generic.go:356] "Generic (PLEG): container finished" podID="6277b932-8a69-4947-afe2-0946a7c6f177" containerID="fec8d6a1326506e9d2303346e455516e0ad98a3b27c19b55905f9a6c2f591199" exitCode=0 Nov 02 18:11:02.500874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:02.495011 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" event={"ID":"6277b932-8a69-4947-afe2-0946a7c6f177","Type":"ContainerDied","Data":"fec8d6a1326506e9d2303346e455516e0ad98a3b27c19b55905f9a6c2f591199"} Nov 02 18:11:02.500874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:02.500315 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" Nov 02 18:11:02.582903 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:02.582833 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6277b932-8a69-4947-afe2-0946a7c6f177-kube-api-access-hqnz6\" (UniqueName: \"kubernetes.io/projected/6277b932-8a69-4947-afe2-0946a7c6f177-kube-api-access-hqnz6\") pod \"6277b932-8a69-4947-afe2-0946a7c6f177\" (UID: \"6277b932-8a69-4947-afe2-0946a7c6f177\") " Nov 02 18:11:02.583548 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:02.582950 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6277b932-8a69-4947-afe2-0946a7c6f177-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6277b932-8a69-4947-afe2-0946a7c6f177-metrics-certs\") pod \"6277b932-8a69-4947-afe2-0946a7c6f177\" (UID: \"6277b932-8a69-4947-afe2-0946a7c6f177\") " Nov 02 18:11:02.583548 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:02.582988 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6277b932-8a69-4947-afe2-0946a7c6f177-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6277b932-8a69-4947-afe2-0946a7c6f177-maas-api-tls\") pod \"6277b932-8a69-4947-afe2-0946a7c6f177\" (UID: \"6277b932-8a69-4947-afe2-0946a7c6f177\") " Nov 02 18:11:02.583548 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:02.583263 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6277b932-8a69-4947-afe2-0946a7c6f177-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6277b932-8a69-4947-afe2-0946a7c6f177-openshift-service-ca\") pod \"6277b932-8a69-4947-afe2-0946a7c6f177\" (UID: \"6277b932-8a69-4947-afe2-0946a7c6f177\") " Nov 02 18:11:02.598237 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:02.584579 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6277b932-8a69-4947-afe2-0946a7c6f177-openshift-service-ca" pod "6277b932-8a69-4947-afe2-0946a7c6f177" (UID: "6277b932-8a69-4947-afe2-0946a7c6f177"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:02.614032 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:02.604650 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6277b932-8a69-4947-afe2-0946a7c6f177-metrics-certs" pod "6277b932-8a69-4947-afe2-0946a7c6f177" (UID: "6277b932-8a69-4947-afe2-0946a7c6f177"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:11:02.614032 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:02.604830 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6277b932-8a69-4947-afe2-0946a7c6f177-kube-api-access-hqnz6" pod "6277b932-8a69-4947-afe2-0946a7c6f177" (UID: "6277b932-8a69-4947-afe2-0946a7c6f177"). InnerVolumeSpecName "kube-api-access-hqnz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:02.614032 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:02.613972 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6277b932-8a69-4947-afe2-0946a7c6f177-maas-api-tls" pod "6277b932-8a69-4947-afe2-0946a7c6f177" (UID: "6277b932-8a69-4947-afe2-0946a7c6f177"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:11:02.734870 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:02.734768 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6277b932-8a69-4947-afe2-0946a7c6f177-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:02.735098 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:02.735075 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqnz6\" (UniqueName: \"kubernetes.io/projected/6277b932-8a69-4947-afe2-0946a7c6f177-kube-api-access-hqnz6\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:02.735236 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:02.735220 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6277b932-8a69-4947-afe2-0946a7c6f177-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:02.735337 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:02.735324 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6277b932-8a69-4947-afe2-0946a7c6f177-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:03.505534 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:03.505506 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" Nov 02 18:11:03.505703 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:03.505567 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb" event={"ID":"6277b932-8a69-4947-afe2-0946a7c6f177","Type":"ContainerDied","Data":"c93d1acfd8b5ed2b4d10971e1bc4f2dd15e2713d846542a79a2b539de24bb577"} Nov 02 18:11:03.505703 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:03.505619 4860 scope.go:122] "RemoveContainer" containerID="fec8d6a1326506e9d2303346e455516e0ad98a3b27c19b55905f9a6c2f591199" Nov 02 18:11:03.564382 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:03.564344 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb"] Nov 02 18:11:03.573052 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:03.573021 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d5b905-79d9dcb678-lc5kb"] Nov 02 18:11:05.501621 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:05.501584 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6277b932-8a69-4947-afe2-0946a7c6f177" path="/var/lib/kubelet/pods/6277b932-8a69-4947-afe2-0946a7c6f177/volumes" Nov 02 18:11:21.772936 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:11:21.772897 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:11:39.091319 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:39.091276 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m"] Nov 02 18:11:39.103006 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:39.102953 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm"] Nov 02 18:11:39.103244 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:39.103181 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" podUID="edbe40ed-0184-44e4-990c-2b7a3fd51f5f" containerName="istio-proxy" containerID="cri-o://0ab71876983a48a4176a21bcd725fc54a43cf3027848dd51e64d85da27fde6ac" gracePeriod=30 Nov 02 18:11:39.103597 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:39.103563 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" podUID="cef174d1-f0e7-4748-997a-fa814937890c" containerName="istio-proxy" containerID="cri-o://04b2682221e9d31acfba1c6db2f8298d1ebbd56c142cbd668430517d23a4d90b" gracePeriod=30 Nov 02 18:11:39.111282 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:39.111255 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5"] Nov 02 18:11:39.111592 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:39.111522 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" podUID="95ef1178-de19-4f53-9646-f4ac3ed52f57" containerName="istio-proxy" containerID="cri-o://416d2d45c71688cdab5ab65c2b44cf698c60c35f49ea83ad91929a47faf6a4e5" gracePeriod=30 Nov 02 18:11:39.781718 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:39.781683 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6"] Nov 02 18:11:39.781977 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:39.781950 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" podUID="6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" containerName="istio-proxy" containerID="cri-o://a0714c31903728472a126cdc344aa293451b24a277b5d4514db7084e125b6af9" gracePeriod=30 Nov 02 18:11:39.886510 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:39.886475 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr"] Nov 02 18:11:39.887472 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:39.887434 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" podUID="e9293d64-b25e-4076-b707-fc65717da6a8" containerName="istio-proxy" containerID="cri-o://4ac94621743bb320f5be5d0c9c03c21a8977f1c6ea50e0e744011f653dc35b5b" gracePeriod=30 Nov 02 18:11:40.061874 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:40.061708 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8"] Nov 02 18:11:40.062068 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:40.062042 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" podUID="1fb27095-a4ff-429a-91be-6f3e28872f18" containerName="istio-proxy" containerID="cri-o://40f12b1a26d758bb57b6b3844dba58f954012ae9ebe1a61242fb8fa54a56cd47" gracePeriod=30 Nov 02 18:11:41.255798 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:41.255761 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:11:41.256236 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:41.255819 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" podUID="edbe40ed-0184-44e4-990c-2b7a3fd51f5f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:11:41.488044 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:41.488002 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:11:41.488214 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:41.488066 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" podUID="95ef1178-de19-4f53-9646-f4ac3ed52f57" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:11:41.566257 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:41.566223 4860 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:11:41.566448 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:41.566281 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" podUID="cef174d1-f0e7-4748-997a-fa814937890c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:11:44.464221 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.464190 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 18:11:44.502950 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.502924 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:11:44.550819 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.550789 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 18:11:44.613611 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.613575 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-credential-socket\") pod \"95ef1178-de19-4f53-9646-f4ac3ed52f57\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " Nov 02 18:11:44.613805 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.613663 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-credential-socket\") pod \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " Nov 02 18:11:44.613805 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.613701 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-podinfo\") pod \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " Nov 02 18:11:44.613805 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.613744 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-data\") pod \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " Nov 02 18:11:44.613805 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.613775 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cef174d1-f0e7-4748-997a-fa814937890c-kube-api-access-rvndd\" (UniqueName: \"kubernetes.io/projected/cef174d1-f0e7-4748-997a-fa814937890c-kube-api-access-rvndd\") pod \"cef174d1-f0e7-4748-997a-fa814937890c\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " Nov 02 18:11:44.613805 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.613805 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-token\" (UniqueName: \"kubernetes.io/projected/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-token\") pod \"95ef1178-de19-4f53-9646-f4ac3ed52f57\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " Nov 02 18:11:44.614122 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.613866 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-credential-socket" pod "95ef1178-de19-4f53-9646-f4ac3ed52f57" (UID: "95ef1178-de19-4f53-9646-f4ac3ed52f57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.614122 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.613877 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-workload-socket\") pod \"95ef1178-de19-4f53-9646-f4ac3ed52f57\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " Nov 02 18:11:44.614122 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.613936 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-podinfo\") pod \"95ef1178-de19-4f53-9646-f4ac3ed52f57\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " Nov 02 18:11:44.614122 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.614025 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-credential-socket" pod "edbe40ed-0184-44e4-990c-2b7a3fd51f5f" (UID: "edbe40ed-0184-44e4-990c-2b7a3fd51f5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.614122 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.614052 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-data\") pod \"95ef1178-de19-4f53-9646-f4ac3ed52f57\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " Nov 02 18:11:44.614122 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.614082 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-token\" (UniqueName: \"kubernetes.io/projected/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-token\") pod \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " Nov 02 18:11:44.614122 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.614116 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-workload-socket" pod "95ef1178-de19-4f53-9646-f4ac3ed52f57" (UID: "95ef1178-de19-4f53-9646-f4ac3ed52f57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.614505 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.614120 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-workload-certs\") pod \"95ef1178-de19-4f53-9646-f4ac3ed52f57\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " Nov 02 18:11:44.614505 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.614177 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cef174d1-f0e7-4748-997a-fa814937890c-istio-token\" (UniqueName: \"kubernetes.io/projected/cef174d1-f0e7-4748-997a-fa814937890c-istio-token\") pod \"cef174d1-f0e7-4748-997a-fa814937890c\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " Nov 02 18:11:44.614505 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.614245 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-workload-certs\") pod \"cef174d1-f0e7-4748-997a-fa814937890c\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " Nov 02 18:11:44.614505 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.614263 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-workload-certs" pod "95ef1178-de19-4f53-9646-f4ac3ed52f57" (UID: "95ef1178-de19-4f53-9646-f4ac3ed52f57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.614505 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.614261 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-data" pod "edbe40ed-0184-44e4-990c-2b7a3fd51f5f" (UID: "edbe40ed-0184-44e4-990c-2b7a3fd51f5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.614505 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.614280 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-istio-data\") pod \"cef174d1-f0e7-4748-997a-fa814937890c\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " Nov 02 18:11:44.614505 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.614314 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-workload-certs\") pod \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " Nov 02 18:11:44.614505 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.614380 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-credential-socket\") pod \"cef174d1-f0e7-4748-997a-fa814937890c\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " Nov 02 18:11:44.614505 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.614413 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ef1178-de19-4f53-9646-f4ac3ed52f57-kube-api-access-g79gx\" (UniqueName: \"kubernetes.io/projected/95ef1178-de19-4f53-9646-f4ac3ed52f57-kube-api-access-g79gx\") pod \"95ef1178-de19-4f53-9646-f4ac3ed52f57\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " Nov 02 18:11:44.614505 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.614458 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-envoy\") pod \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " Nov 02 18:11:44.615080 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.614547 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-workload-socket\") pod \"cef174d1-f0e7-4748-997a-fa814937890c\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " Nov 02 18:11:44.615080 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.614624 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-workload-certs" pod "cef174d1-f0e7-4748-997a-fa814937890c" (UID: "cef174d1-f0e7-4748-997a-fa814937890c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.615080 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.614823 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-data" pod "95ef1178-de19-4f53-9646-f4ac3ed52f57" (UID: "95ef1178-de19-4f53-9646-f4ac3ed52f57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.615080 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.615066 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-istio-data" pod "cef174d1-f0e7-4748-997a-fa814937890c" (UID: "cef174d1-f0e7-4748-997a-fa814937890c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.615294 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.615210 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-workload-certs" pod "edbe40ed-0184-44e4-990c-2b7a3fd51f5f" (UID: "edbe40ed-0184-44e4-990c-2b7a3fd51f5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.615383 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.615356 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-credential-socket" pod "cef174d1-f0e7-4748-997a-fa814937890c" (UID: "cef174d1-f0e7-4748-997a-fa814937890c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.616150 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.615506 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-workload-socket\") pod \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " Nov 02 18:11:44.616150 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.615562 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cef174d1-f0e7-4748-997a-fa814937890c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef174d1-f0e7-4748-997a-fa814937890c-istiod-ca-cert\") pod \"cef174d1-f0e7-4748-997a-fa814937890c\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " Nov 02 18:11:44.616150 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.615603 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95ef1178-de19-4f53-9646-f4ac3ed52f57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ef1178-de19-4f53-9646-f4ac3ed52f57-istiod-ca-cert\") pod \"95ef1178-de19-4f53-9646-f4ac3ed52f57\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " Nov 02 18:11:44.616150 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.615892 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-kube-api-access-h7fxt\" (UniqueName: \"kubernetes.io/projected/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-kube-api-access-h7fxt\") pod \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " Nov 02 18:11:44.616150 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.615971 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-workload-socket" pod "edbe40ed-0184-44e4-990c-2b7a3fd51f5f" (UID: "edbe40ed-0184-44e4-990c-2b7a3fd51f5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.616150 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.615992 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-istio-envoy\") pod \"cef174d1-f0e7-4748-997a-fa814937890c\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " Nov 02 18:11:44.616150 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.616087 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cef174d1-f0e7-4748-997a-fa814937890c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef174d1-f0e7-4748-997a-fa814937890c-istio-podinfo\") pod \"cef174d1-f0e7-4748-997a-fa814937890c\" (UID: \"cef174d1-f0e7-4748-997a-fa814937890c\") " Nov 02 18:11:44.616150 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.616133 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istiod-ca-cert\") pod \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\" (UID: \"edbe40ed-0184-44e4-990c-2b7a3fd51f5f\") " Nov 02 18:11:44.616617 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.616161 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-workload-socket" pod "cef174d1-f0e7-4748-997a-fa814937890c" (UID: "cef174d1-f0e7-4748-997a-fa814937890c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.616617 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.616166 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-envoy\") pod \"95ef1178-de19-4f53-9646-f4ac3ed52f57\" (UID: \"95ef1178-de19-4f53-9646-f4ac3ed52f57\") " Nov 02 18:11:44.616617 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.616460 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef174d1-f0e7-4748-997a-fa814937890c-istiod-ca-cert" pod "cef174d1-f0e7-4748-997a-fa814937890c" (UID: "cef174d1-f0e7-4748-997a-fa814937890c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:44.619885 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.617601 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.619885 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.617630 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.619885 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.617648 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.619885 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.617664 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.619885 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.617678 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.619885 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.617693 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.619885 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.617706 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.619885 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.617719 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.619885 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.617734 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.619885 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.617747 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.619885 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.617761 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.619885 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.617776 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.619885 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.617790 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef174d1-f0e7-4748-997a-fa814937890c-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.619885 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.618141 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95ef1178-de19-4f53-9646-f4ac3ed52f57-istiod-ca-cert" pod "95ef1178-de19-4f53-9646-f4ac3ed52f57" (UID: "95ef1178-de19-4f53-9646-f4ac3ed52f57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:44.621678 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.621640 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istiod-ca-cert" pod "edbe40ed-0184-44e4-990c-2b7a3fd51f5f" (UID: "edbe40ed-0184-44e4-990c-2b7a3fd51f5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:44.623068 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.622990 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cef174d1-f0e7-4748-997a-fa814937890c-istio-podinfo" pod "cef174d1-f0e7-4748-997a-fa814937890c" (UID: "cef174d1-f0e7-4748-997a-fa814937890c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:44.626670 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.626632 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-istio-envoy" pod "cef174d1-f0e7-4748-997a-fa814937890c" (UID: "cef174d1-f0e7-4748-997a-fa814937890c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.629670 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.629636 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-kube-api-access-h7fxt" pod "edbe40ed-0184-44e4-990c-2b7a3fd51f5f" (UID: "edbe40ed-0184-44e4-990c-2b7a3fd51f5f"). InnerVolumeSpecName "kube-api-access-h7fxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:44.630370 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.630216 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-podinfo" pod "edbe40ed-0184-44e4-990c-2b7a3fd51f5f" (UID: "edbe40ed-0184-44e4-990c-2b7a3fd51f5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:44.630370 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.630325 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef174d1-f0e7-4748-997a-fa814937890c-istio-token" pod "cef174d1-f0e7-4748-997a-fa814937890c" (UID: "cef174d1-f0e7-4748-997a-fa814937890c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:44.630541 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.630369 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-token" pod "edbe40ed-0184-44e4-990c-2b7a3fd51f5f" (UID: "edbe40ed-0184-44e4-990c-2b7a3fd51f5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:44.630541 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.630405 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ef1178-de19-4f53-9646-f4ac3ed52f57-kube-api-access-g79gx" pod "95ef1178-de19-4f53-9646-f4ac3ed52f57" (UID: "95ef1178-de19-4f53-9646-f4ac3ed52f57"). InnerVolumeSpecName "kube-api-access-g79gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:44.630836 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.630811 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-token" pod "95ef1178-de19-4f53-9646-f4ac3ed52f57" (UID: "95ef1178-de19-4f53-9646-f4ac3ed52f57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:44.630972 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.630865 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef174d1-f0e7-4748-997a-fa814937890c-kube-api-access-rvndd" pod "cef174d1-f0e7-4748-997a-fa814937890c" (UID: "cef174d1-f0e7-4748-997a-fa814937890c"). InnerVolumeSpecName "kube-api-access-rvndd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:44.631048 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.630984 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-podinfo" pod "95ef1178-de19-4f53-9646-f4ac3ed52f57" (UID: "95ef1178-de19-4f53-9646-f4ac3ed52f57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:44.633287 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.633223 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-envoy" pod "95ef1178-de19-4f53-9646-f4ac3ed52f57" (UID: "95ef1178-de19-4f53-9646-f4ac3ed52f57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.633287 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.633249 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-envoy" pod "edbe40ed-0184-44e4-990c-2b7a3fd51f5f" (UID: "edbe40ed-0184-44e4-990c-2b7a3fd51f5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:44.719331 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.719243 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.719331 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.719282 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ef1178-de19-4f53-9646-f4ac3ed52f57-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.719331 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.719298 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7fxt\" (UniqueName: \"kubernetes.io/projected/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-kube-api-access-h7fxt\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.719331 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.719314 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef174d1-f0e7-4748-997a-fa814937890c-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.719331 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.719328 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef174d1-f0e7-4748-997a-fa814937890c-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.719701 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.719340 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.719701 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.719355 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.719701 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.719368 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.719701 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.719381 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvndd\" (UniqueName: \"kubernetes.io/projected/cef174d1-f0e7-4748-997a-fa814937890c-kube-api-access-rvndd\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.719701 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.719396 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.719701 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.719408 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ef1178-de19-4f53-9646-f4ac3ed52f57-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.719701 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.719420 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edbe40ed-0184-44e4-990c-2b7a3fd51f5f-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.719701 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.719433 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef174d1-f0e7-4748-997a-fa814937890c-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:44.719701 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:44.719446 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g79gx\" (UniqueName: \"kubernetes.io/projected/95ef1178-de19-4f53-9646-f4ac3ed52f57-kube-api-access-g79gx\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.086774 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.086519 4860 generic.go:356] "Generic (PLEG): container finished" podID="95ef1178-de19-4f53-9646-f4ac3ed52f57" containerID="416d2d45c71688cdab5ab65c2b44cf698c60c35f49ea83ad91929a47faf6a4e5" exitCode=0 Nov 02 18:11:45.087183 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.087134 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" event={"ID":"95ef1178-de19-4f53-9646-f4ac3ed52f57","Type":"ContainerDied","Data":"416d2d45c71688cdab5ab65c2b44cf698c60c35f49ea83ad91929a47faf6a4e5"} Nov 02 18:11:45.087291 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.087194 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" event={"ID":"95ef1178-de19-4f53-9646-f4ac3ed52f57","Type":"ContainerDied","Data":"2a29ed4c926ee4f10e6cd7cb0b7ead348a52dc5213282580d792463319075fa8"} Nov 02 18:11:45.087291 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.087225 4860 scope.go:122] "RemoveContainer" containerID="416d2d45c71688cdab5ab65c2b44cf698c60c35f49ea83ad91929a47faf6a4e5" Nov 02 18:11:45.087461 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.087447 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5" Nov 02 18:11:45.099159 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.098942 4860 generic.go:356] "Generic (PLEG): container finished" podID="6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" containerID="a0714c31903728472a126cdc344aa293451b24a277b5d4514db7084e125b6af9" exitCode=0 Nov 02 18:11:45.099159 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.099081 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" event={"ID":"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89","Type":"ContainerDied","Data":"a0714c31903728472a126cdc344aa293451b24a277b5d4514db7084e125b6af9"} Nov 02 18:11:45.102872 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.101590 4860 generic.go:356] "Generic (PLEG): container finished" podID="edbe40ed-0184-44e4-990c-2b7a3fd51f5f" containerID="0ab71876983a48a4176a21bcd725fc54a43cf3027848dd51e64d85da27fde6ac" exitCode=0 Nov 02 18:11:45.102872 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.101694 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" event={"ID":"edbe40ed-0184-44e4-990c-2b7a3fd51f5f","Type":"ContainerDied","Data":"0ab71876983a48a4176a21bcd725fc54a43cf3027848dd51e64d85da27fde6ac"} Nov 02 18:11:45.102872 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.101741 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" event={"ID":"edbe40ed-0184-44e4-990c-2b7a3fd51f5f","Type":"ContainerDied","Data":"05bc174f5be43f8bc5a86bd8acce1e47b453d26509954953b46597059ee69525"} Nov 02 18:11:45.102872 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.101830 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm" Nov 02 18:11:45.106562 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.106538 4860 generic.go:356] "Generic (PLEG): container finished" podID="cef174d1-f0e7-4748-997a-fa814937890c" containerID="04b2682221e9d31acfba1c6db2f8298d1ebbd56c142cbd668430517d23a4d90b" exitCode=0 Nov 02 18:11:45.106880 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.106711 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" event={"ID":"cef174d1-f0e7-4748-997a-fa814937890c","Type":"ContainerDied","Data":"04b2682221e9d31acfba1c6db2f8298d1ebbd56c142cbd668430517d23a4d90b"} Nov 02 18:11:45.107066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.107043 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" event={"ID":"cef174d1-f0e7-4748-997a-fa814937890c","Type":"ContainerDied","Data":"9f4fb8609053c2db065f0d53c6302af7ebe67229a0e89135d34115f421a7c11f"} Nov 02 18:11:45.107066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.106796 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m" Nov 02 18:11:45.143969 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.141266 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 18:11:45.235876 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.231334 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istiod-ca-cert\") pod \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " Nov 02 18:11:45.235876 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.231410 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-podinfo\") pod \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " Nov 02 18:11:45.235876 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.231475 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-workload-socket\") pod \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " Nov 02 18:11:45.235876 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.231501 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-credential-socket\") pod \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " Nov 02 18:11:45.235876 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.231759 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-kube-api-access-jrcct\" (UniqueName: \"kubernetes.io/projected/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-kube-api-access-jrcct\") pod \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " Nov 02 18:11:45.235876 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.231806 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-workload-certs\") pod \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " Nov 02 18:11:45.235876 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.231875 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-token\" (UniqueName: \"kubernetes.io/projected/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-token\") pod \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " Nov 02 18:11:45.235876 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.231920 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-data\") pod \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " Nov 02 18:11:45.235876 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.231968 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-envoy\") pod \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\" (UID: \"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89\") " Nov 02 18:11:45.290442 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.249110 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istiod-ca-cert" pod "6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" (UID: "6975e1bd-ced6-4be9-8808-2e6fa6ff2a89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:45.290442 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.249401 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-data" pod "6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" (UID: "6975e1bd-ced6-4be9-8808-2e6fa6ff2a89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.290442 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.249575 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-workload-socket" pod "6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" (UID: "6975e1bd-ced6-4be9-8808-2e6fa6ff2a89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.290442 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.249928 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-workload-certs" pod "6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" (UID: "6975e1bd-ced6-4be9-8808-2e6fa6ff2a89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.290442 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.250948 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-credential-socket" pod "6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" (UID: "6975e1bd-ced6-4be9-8808-2e6fa6ff2a89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.290442 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.287393 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-podinfo" pod "6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" (UID: "6975e1bd-ced6-4be9-8808-2e6fa6ff2a89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:45.291218 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.291163 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-token" pod "6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" (UID: "6975e1bd-ced6-4be9-8808-2e6fa6ff2a89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:45.299625 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.299586 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-envoy" pod "6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" (UID: "6975e1bd-ced6-4be9-8808-2e6fa6ff2a89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.299833 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.299804 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-kube-api-access-jrcct" pod "6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" (UID: "6975e1bd-ced6-4be9-8808-2e6fa6ff2a89"). InnerVolumeSpecName "kube-api-access-jrcct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:45.321595 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.321569 4860 scope.go:122] "RemoveContainer" containerID="416d2d45c71688cdab5ab65c2b44cf698c60c35f49ea83ad91929a47faf6a4e5" Nov 02 18:11:45.323943 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:11:45.322486 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416d2d45c71688cdab5ab65c2b44cf698c60c35f49ea83ad91929a47faf6a4e5\": container with ID starting with 416d2d45c71688cdab5ab65c2b44cf698c60c35f49ea83ad91929a47faf6a4e5 not found: ID does not exist" containerID="416d2d45c71688cdab5ab65c2b44cf698c60c35f49ea83ad91929a47faf6a4e5" Nov 02 18:11:45.323943 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.322529 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416d2d45c71688cdab5ab65c2b44cf698c60c35f49ea83ad91929a47faf6a4e5"} err="failed to get container status \"416d2d45c71688cdab5ab65c2b44cf698c60c35f49ea83ad91929a47faf6a4e5\": rpc error: code = NotFound desc = could not find container \"416d2d45c71688cdab5ab65c2b44cf698c60c35f49ea83ad91929a47faf6a4e5\": container with ID starting with 416d2d45c71688cdab5ab65c2b44cf698c60c35f49ea83ad91929a47faf6a4e5 not found: ID does not exist" Nov 02 18:11:45.323943 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.322552 4860 scope.go:122] "RemoveContainer" containerID="0ab71876983a48a4176a21bcd725fc54a43cf3027848dd51e64d85da27fde6ac" Nov 02 18:11:45.328461 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.328379 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m"] Nov 02 18:11:45.335448 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.335240 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrcct\" (UniqueName: \"kubernetes.io/projected/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-kube-api-access-jrcct\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.335448 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.335268 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.335448 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.335284 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.335448 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.335297 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.335448 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.335309 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.335448 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.335322 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.335448 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.335335 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.335448 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.335350 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.335448 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.335367 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.336179 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.336152 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-qrd5m"] Nov 02 18:11:45.347715 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.347424 4860 scope.go:122] "RemoveContainer" containerID="0ab71876983a48a4176a21bcd725fc54a43cf3027848dd51e64d85da27fde6ac" Nov 02 18:11:45.348333 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:11:45.347927 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ab71876983a48a4176a21bcd725fc54a43cf3027848dd51e64d85da27fde6ac\": container with ID starting with 0ab71876983a48a4176a21bcd725fc54a43cf3027848dd51e64d85da27fde6ac not found: ID does not exist" containerID="0ab71876983a48a4176a21bcd725fc54a43cf3027848dd51e64d85da27fde6ac" Nov 02 18:11:45.348333 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.347964 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ab71876983a48a4176a21bcd725fc54a43cf3027848dd51e64d85da27fde6ac"} err="failed to get container status \"0ab71876983a48a4176a21bcd725fc54a43cf3027848dd51e64d85da27fde6ac\": rpc error: code = NotFound desc = could not find container \"0ab71876983a48a4176a21bcd725fc54a43cf3027848dd51e64d85da27fde6ac\": container with ID starting with 0ab71876983a48a4176a21bcd725fc54a43cf3027848dd51e64d85da27fde6ac not found: ID does not exist" Nov 02 18:11:45.348333 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.347990 4860 scope.go:122] "RemoveContainer" containerID="04b2682221e9d31acfba1c6db2f8298d1ebbd56c142cbd668430517d23a4d90b" Nov 02 18:11:45.349381 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.349321 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5"] Nov 02 18:11:45.356205 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.356064 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-f7np5"] Nov 02 18:11:45.367489 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.367197 4860 scope.go:122] "RemoveContainer" containerID="04b2682221e9d31acfba1c6db2f8298d1ebbd56c142cbd668430517d23a4d90b" Nov 02 18:11:45.367760 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:11:45.367664 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04b2682221e9d31acfba1c6db2f8298d1ebbd56c142cbd668430517d23a4d90b\": container with ID starting with 04b2682221e9d31acfba1c6db2f8298d1ebbd56c142cbd668430517d23a4d90b not found: ID does not exist" containerID="04b2682221e9d31acfba1c6db2f8298d1ebbd56c142cbd668430517d23a4d90b" Nov 02 18:11:45.367760 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.367698 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b2682221e9d31acfba1c6db2f8298d1ebbd56c142cbd668430517d23a4d90b"} err="failed to get container status \"04b2682221e9d31acfba1c6db2f8298d1ebbd56c142cbd668430517d23a4d90b\": rpc error: code = NotFound desc = could not find container \"04b2682221e9d31acfba1c6db2f8298d1ebbd56c142cbd668430517d23a4d90b\": container with ID starting with 04b2682221e9d31acfba1c6db2f8298d1ebbd56c142cbd668430517d23a4d90b not found: ID does not exist" Nov 02 18:11:45.414599 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.414563 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm"] Nov 02 18:11:45.416765 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.416722 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-2phkm"] Nov 02 18:11:45.508243 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.508212 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95ef1178-de19-4f53-9646-f4ac3ed52f57" path="/var/lib/kubelet/pods/95ef1178-de19-4f53-9646-f4ac3ed52f57/volumes" Nov 02 18:11:45.509493 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.509468 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cef174d1-f0e7-4748-997a-fa814937890c" path="/var/lib/kubelet/pods/cef174d1-f0e7-4748-997a-fa814937890c/volumes" Nov 02 18:11:45.510264 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.510237 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edbe40ed-0184-44e4-990c-2b7a3fd51f5f" path="/var/lib/kubelet/pods/edbe40ed-0184-44e4-990c-2b7a3fd51f5f/volumes" Nov 02 18:11:45.517116 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.517092 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 18:11:45.567428 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.566603 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:11:45.649124 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.649035 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-podinfo\") pod \"1fb27095-a4ff-429a-91be-6f3e28872f18\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " Nov 02 18:11:45.649436 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.649403 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9293d64-b25e-4076-b707-fc65717da6a8-istio-token\" (UniqueName: \"kubernetes.io/projected/e9293d64-b25e-4076-b707-fc65717da6a8-istio-token\") pod \"e9293d64-b25e-4076-b707-fc65717da6a8\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " Nov 02 18:11:45.649568 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.649460 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-credential-socket\") pod \"e9293d64-b25e-4076-b707-fc65717da6a8\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " Nov 02 18:11:45.649568 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.649504 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-data\") pod \"1fb27095-a4ff-429a-91be-6f3e28872f18\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " Nov 02 18:11:45.649568 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.649544 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-istio-data\") pod \"e9293d64-b25e-4076-b707-fc65717da6a8\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " Nov 02 18:11:45.649756 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.649583 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9293d64-b25e-4076-b707-fc65717da6a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9293d64-b25e-4076-b707-fc65717da6a8-istio-podinfo\") pod \"e9293d64-b25e-4076-b707-fc65717da6a8\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " Nov 02 18:11:45.649756 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.649609 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fb27095-a4ff-429a-91be-6f3e28872f18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb27095-a4ff-429a-91be-6f3e28872f18-istiod-ca-cert\") pod \"1fb27095-a4ff-429a-91be-6f3e28872f18\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " Nov 02 18:11:45.649756 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.649657 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb27095-a4ff-429a-91be-6f3e28872f18-kube-api-access-8qvgn\" (UniqueName: \"kubernetes.io/projected/1fb27095-a4ff-429a-91be-6f3e28872f18-kube-api-access-8qvgn\") pod \"1fb27095-a4ff-429a-91be-6f3e28872f18\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " Nov 02 18:11:45.649756 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.649732 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-credential-socket\") pod \"1fb27095-a4ff-429a-91be-6f3e28872f18\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " Nov 02 18:11:45.650030 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.649760 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-workload-certs\") pod \"1fb27095-a4ff-429a-91be-6f3e28872f18\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " Nov 02 18:11:45.650030 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.649786 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-istio-envoy\") pod \"e9293d64-b25e-4076-b707-fc65717da6a8\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " Nov 02 18:11:45.650030 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.649807 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-workload-socket\") pod \"1fb27095-a4ff-429a-91be-6f3e28872f18\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " Nov 02 18:11:45.650030 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.649875 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9293d64-b25e-4076-b707-fc65717da6a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9293d64-b25e-4076-b707-fc65717da6a8-istiod-ca-cert\") pod \"e9293d64-b25e-4076-b707-fc65717da6a8\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " Nov 02 18:11:45.650030 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.649927 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-envoy\") pod \"1fb27095-a4ff-429a-91be-6f3e28872f18\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " Nov 02 18:11:45.650030 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.649964 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-workload-certs\") pod \"e9293d64-b25e-4076-b707-fc65717da6a8\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " Nov 02 18:11:45.650030 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.650007 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9293d64-b25e-4076-b707-fc65717da6a8-kube-api-access-rdwvn\" (UniqueName: \"kubernetes.io/projected/e9293d64-b25e-4076-b707-fc65717da6a8-kube-api-access-rdwvn\") pod \"e9293d64-b25e-4076-b707-fc65717da6a8\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " Nov 02 18:11:45.650395 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.650047 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-token\" (UniqueName: \"kubernetes.io/projected/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-token\") pod \"1fb27095-a4ff-429a-91be-6f3e28872f18\" (UID: \"1fb27095-a4ff-429a-91be-6f3e28872f18\") " Nov 02 18:11:45.650395 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.650064 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-workload-socket\") pod \"e9293d64-b25e-4076-b707-fc65717da6a8\" (UID: \"e9293d64-b25e-4076-b707-fc65717da6a8\") " Nov 02 18:11:45.650395 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.650065 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-istio-data" pod "e9293d64-b25e-4076-b707-fc65717da6a8" (UID: "e9293d64-b25e-4076-b707-fc65717da6a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.650778 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.650682 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.650943 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.650909 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-data" pod "1fb27095-a4ff-429a-91be-6f3e28872f18" (UID: "1fb27095-a4ff-429a-91be-6f3e28872f18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.651528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.651502 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-credential-socket" pod "1fb27095-a4ff-429a-91be-6f3e28872f18" (UID: "1fb27095-a4ff-429a-91be-6f3e28872f18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.651629 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.651527 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-workload-certs" pod "e9293d64-b25e-4076-b707-fc65717da6a8" (UID: "e9293d64-b25e-4076-b707-fc65717da6a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.651629 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.651553 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-workload-certs" pod "1fb27095-a4ff-429a-91be-6f3e28872f18" (UID: "1fb27095-a4ff-429a-91be-6f3e28872f18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.652282 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.651979 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-workload-socket" pod "e9293d64-b25e-4076-b707-fc65717da6a8" (UID: "e9293d64-b25e-4076-b707-fc65717da6a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.652382 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.652327 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-credential-socket" pod "e9293d64-b25e-4076-b707-fc65717da6a8" (UID: "e9293d64-b25e-4076-b707-fc65717da6a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.653207 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.653179 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb27095-a4ff-429a-91be-6f3e28872f18-istiod-ca-cert" pod "1fb27095-a4ff-429a-91be-6f3e28872f18" (UID: "1fb27095-a4ff-429a-91be-6f3e28872f18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:45.653492 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.653467 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-workload-socket" pod "1fb27095-a4ff-429a-91be-6f3e28872f18" (UID: "1fb27095-a4ff-429a-91be-6f3e28872f18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.653734 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.653709 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9293d64-b25e-4076-b707-fc65717da6a8-istiod-ca-cert" pod "e9293d64-b25e-4076-b707-fc65717da6a8" (UID: "e9293d64-b25e-4076-b707-fc65717da6a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:45.656997 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.656968 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-istio-envoy" pod "e9293d64-b25e-4076-b707-fc65717da6a8" (UID: "e9293d64-b25e-4076-b707-fc65717da6a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.658243 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.658216 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-token" pod "1fb27095-a4ff-429a-91be-6f3e28872f18" (UID: "1fb27095-a4ff-429a-91be-6f3e28872f18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:45.658822 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.658795 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9293d64-b25e-4076-b707-fc65717da6a8-kube-api-access-rdwvn" pod "e9293d64-b25e-4076-b707-fc65717da6a8" (UID: "e9293d64-b25e-4076-b707-fc65717da6a8"). InnerVolumeSpecName "kube-api-access-rdwvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:45.659136 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.659110 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-envoy" pod "1fb27095-a4ff-429a-91be-6f3e28872f18" (UID: "1fb27095-a4ff-429a-91be-6f3e28872f18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:45.659136 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.659132 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-podinfo" pod "1fb27095-a4ff-429a-91be-6f3e28872f18" (UID: "1fb27095-a4ff-429a-91be-6f3e28872f18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:45.659317 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.659173 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9293d64-b25e-4076-b707-fc65717da6a8-istio-token" pod "e9293d64-b25e-4076-b707-fc65717da6a8" (UID: "e9293d64-b25e-4076-b707-fc65717da6a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:45.662073 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.662047 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb27095-a4ff-429a-91be-6f3e28872f18-kube-api-access-8qvgn" pod "1fb27095-a4ff-429a-91be-6f3e28872f18" (UID: "1fb27095-a4ff-429a-91be-6f3e28872f18"). InnerVolumeSpecName "kube-api-access-8qvgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:45.667833 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.667811 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9293d64-b25e-4076-b707-fc65717da6a8-istio-podinfo" pod "e9293d64-b25e-4076-b707-fc65717da6a8" (UID: "e9293d64-b25e-4076-b707-fc65717da6a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:45.773622 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.772908 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.773622 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.772960 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.773622 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.772977 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdwvn\" (UniqueName: \"kubernetes.io/projected/e9293d64-b25e-4076-b707-fc65717da6a8-kube-api-access-rdwvn\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.773622 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.772995 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.773622 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.773010 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.773622 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.773023 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.773622 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.773036 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9293d64-b25e-4076-b707-fc65717da6a8-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.773622 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.773050 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.773622 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.773062 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.773622 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.773075 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9293d64-b25e-4076-b707-fc65717da6a8-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.773622 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.773088 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb27095-a4ff-429a-91be-6f3e28872f18-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.773622 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.773102 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qvgn\" (UniqueName: \"kubernetes.io/projected/1fb27095-a4ff-429a-91be-6f3e28872f18-kube-api-access-8qvgn\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.773622 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.773115 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.773622 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.773129 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.773622 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.773142 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9293d64-b25e-4076-b707-fc65717da6a8-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.773622 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.773157 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb27095-a4ff-429a-91be-6f3e28872f18-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:45.773622 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:45.773171 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9293d64-b25e-4076-b707-fc65717da6a8-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:46.122469 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.122436 4860 generic.go:356] "Generic (PLEG): container finished" podID="e9293d64-b25e-4076-b707-fc65717da6a8" containerID="4ac94621743bb320f5be5d0c9c03c21a8977f1c6ea50e0e744011f653dc35b5b" exitCode=0 Nov 02 18:11:46.122671 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.122487 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" Nov 02 18:11:46.122671 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.122529 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" event={"ID":"e9293d64-b25e-4076-b707-fc65717da6a8","Type":"ContainerDied","Data":"4ac94621743bb320f5be5d0c9c03c21a8977f1c6ea50e0e744011f653dc35b5b"} Nov 02 18:11:46.122671 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.122587 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr" event={"ID":"e9293d64-b25e-4076-b707-fc65717da6a8","Type":"ContainerDied","Data":"36e05a6ae1db066818a27a37c26afed9a4d9d20fd79ff7d5d6218cdc68c1c517"} Nov 02 18:11:46.122671 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.122617 4860 scope.go:122] "RemoveContainer" containerID="4ac94621743bb320f5be5d0c9c03c21a8977f1c6ea50e0e744011f653dc35b5b" Nov 02 18:11:46.124769 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.124741 4860 generic.go:356] "Generic (PLEG): container finished" podID="1fb27095-a4ff-429a-91be-6f3e28872f18" containerID="40f12b1a26d758bb57b6b3844dba58f954012ae9ebe1a61242fb8fa54a56cd47" exitCode=0 Nov 02 18:11:46.124913 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.124830 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" Nov 02 18:11:46.125312 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.125281 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" event={"ID":"1fb27095-a4ff-429a-91be-6f3e28872f18","Type":"ContainerDied","Data":"40f12b1a26d758bb57b6b3844dba58f954012ae9ebe1a61242fb8fa54a56cd47"} Nov 02 18:11:46.125429 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.125343 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8" event={"ID":"1fb27095-a4ff-429a-91be-6f3e28872f18","Type":"ContainerDied","Data":"02157eed7e604d68113fa237257af267874fbc2b0b68302c4357f3e2d3f00481"} Nov 02 18:11:46.128594 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.126774 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" Nov 02 18:11:46.128594 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.126801 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6" event={"ID":"6975e1bd-ced6-4be9-8808-2e6fa6ff2a89","Type":"ContainerDied","Data":"f4abcfd600783a8e0ea1f0a966d4510e0d024c036eace473d752662bc7f0c62f"} Nov 02 18:11:46.174410 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.174375 4860 scope.go:122] "RemoveContainer" containerID="4ac94621743bb320f5be5d0c9c03c21a8977f1c6ea50e0e744011f653dc35b5b" Nov 02 18:11:46.174934 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:11:46.174897 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac94621743bb320f5be5d0c9c03c21a8977f1c6ea50e0e744011f653dc35b5b\": container with ID starting with 4ac94621743bb320f5be5d0c9c03c21a8977f1c6ea50e0e744011f653dc35b5b not found: ID does not exist" containerID="4ac94621743bb320f5be5d0c9c03c21a8977f1c6ea50e0e744011f653dc35b5b" Nov 02 18:11:46.175121 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.174944 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac94621743bb320f5be5d0c9c03c21a8977f1c6ea50e0e744011f653dc35b5b"} err="failed to get container status \"4ac94621743bb320f5be5d0c9c03c21a8977f1c6ea50e0e744011f653dc35b5b\": rpc error: code = NotFound desc = could not find container \"4ac94621743bb320f5be5d0c9c03c21a8977f1c6ea50e0e744011f653dc35b5b\": container with ID starting with 4ac94621743bb320f5be5d0c9c03c21a8977f1c6ea50e0e744011f653dc35b5b not found: ID does not exist" Nov 02 18:11:46.175121 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.174966 4860 scope.go:122] "RemoveContainer" containerID="40f12b1a26d758bb57b6b3844dba58f954012ae9ebe1a61242fb8fa54a56cd47" Nov 02 18:11:46.191091 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.191067 4860 scope.go:122] "RemoveContainer" containerID="40f12b1a26d758bb57b6b3844dba58f954012ae9ebe1a61242fb8fa54a56cd47" Nov 02 18:11:46.191378 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:11:46.191362 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f12b1a26d758bb57b6b3844dba58f954012ae9ebe1a61242fb8fa54a56cd47\": container with ID starting with 40f12b1a26d758bb57b6b3844dba58f954012ae9ebe1a61242fb8fa54a56cd47 not found: ID does not exist" containerID="40f12b1a26d758bb57b6b3844dba58f954012ae9ebe1a61242fb8fa54a56cd47" Nov 02 18:11:46.191420 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.191385 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f12b1a26d758bb57b6b3844dba58f954012ae9ebe1a61242fb8fa54a56cd47"} err="failed to get container status \"40f12b1a26d758bb57b6b3844dba58f954012ae9ebe1a61242fb8fa54a56cd47\": rpc error: code = NotFound desc = could not find container \"40f12b1a26d758bb57b6b3844dba58f954012ae9ebe1a61242fb8fa54a56cd47\": container with ID starting with 40f12b1a26d758bb57b6b3844dba58f954012ae9ebe1a61242fb8fa54a56cd47 not found: ID does not exist" Nov 02 18:11:46.191420 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.191401 4860 scope.go:122] "RemoveContainer" containerID="a0714c31903728472a126cdc344aa293451b24a277b5d4514db7084e125b6af9" Nov 02 18:11:46.215202 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.215066 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6"] Nov 02 18:11:46.227509 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.227477 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-prpc6"] Nov 02 18:11:46.279182 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.279150 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr"] Nov 02 18:11:46.283425 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.283396 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-6xlmr"] Nov 02 18:11:46.317692 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.317659 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8"] Nov 02 18:11:46.322523 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.322498 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-bv9b8"] Nov 02 18:11:46.673707 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.673669 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb"] Nov 02 18:11:46.751222 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.751186 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm"] Nov 02 18:11:46.751451 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.751425 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" podUID="dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" containerName="istio-proxy" containerID="cri-o://ff641b2cf9a1dd6b53442dd633731a641f214cafc78ca1cd3fa26e8ac1693ae0" gracePeriod=30 Nov 02 18:11:46.792460 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.792428 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz"] Nov 02 18:11:46.792958 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.792919 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" podUID="9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" containerName="istio-proxy" containerID="cri-o://9c789f0f324a1d3863d433a4bd09ad474e0df306700e59720602436ef13fc6ff" gracePeriod=30 Nov 02 18:11:46.825498 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.825457 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5"] Nov 02 18:11:46.825719 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.825691 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" podUID="10a632f9-0197-4f8c-b414-c15b29caea6e" containerName="discovery" containerID="cri-o://b2e39f38ce529a4f34acf22b28591824ea70d6c4ad384d6e00d423dfccb5cf5a" gracePeriod=30 Nov 02 18:11:46.871014 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.870972 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v"] Nov 02 18:11:46.871452 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.871417 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" podUID="62b05251-7074-41bc-8994-c1051abee475" containerName="istio-proxy" containerID="cri-o://5a4a2440f0ea022d0c762cefdfcaa529ea8b44e559e079baffd8fd191a54c120" gracePeriod=30 Nov 02 18:11:46.876284 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.876258 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh"] Nov 02 18:11:46.876516 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.876468 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" podUID="c92d9c87-7ba0-4d6e-b659-9e916730c79e" containerName="istio-proxy" containerID="cri-o://71d866d15c5ebd732f0d33a1bda18a78921026df493328a1a5e9f5fab9f1a050" gracePeriod=30 Nov 02 18:11:46.930357 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.929753 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf"] Nov 02 18:11:46.930357 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:46.930032 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" podUID="b53fb7b2-d91c-409b-bf83-e24b95a7608f" containerName="istio-proxy" containerID="cri-o://30faeedd7646bee6bf8c2c08acf678ef95aee941ff9af52570e2eb64031430ad" gracePeriod=30 Nov 02 18:11:47.141771 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.141732 4860 generic.go:356] "Generic (PLEG): container finished" podID="10a632f9-0197-4f8c-b414-c15b29caea6e" containerID="b2e39f38ce529a4f34acf22b28591824ea70d6c4ad384d6e00d423dfccb5cf5a" exitCode=0 Nov 02 18:11:47.141995 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.141820 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" event={"ID":"10a632f9-0197-4f8c-b414-c15b29caea6e","Type":"ContainerDied","Data":"b2e39f38ce529a4f34acf22b28591824ea70d6c4ad384d6e00d423dfccb5cf5a"} Nov 02 18:11:47.146689 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.146657 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" podUID="e31cdb01-9f71-4eae-ad26-6f1e19915ffd" containerName="istio-proxy" containerID="cri-o://9da9acd413ede11a467a5cf900fff9debafc3b5dce5118d9af0863fbd69f080d" gracePeriod=30 Nov 02 18:11:47.186922 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.186838 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 18:11:47.198070 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.198042 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-token\" (UniqueName: \"kubernetes.io/projected/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-token\") pod \"10a632f9-0197-4f8c-b414-c15b29caea6e\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " Nov 02 18:11:47.198243 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.198100 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a632f9-0197-4f8c-b414-c15b29caea6e-kube-api-access-whsl4\" (UniqueName: \"kubernetes.io/projected/10a632f9-0197-4f8c-b414-c15b29caea6e-kube-api-access-whsl4\") pod \"10a632f9-0197-4f8c-b414-c15b29caea6e\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " Nov 02 18:11:47.198243 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.198172 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a632f9-0197-4f8c-b414-c15b29caea6e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a632f9-0197-4f8c-b414-c15b29caea6e-local-certs\") pod \"10a632f9-0197-4f8c-b414-c15b29caea6e\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " Nov 02 18:11:47.198377 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.198321 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-csr-ca-configmap\") pod \"10a632f9-0197-4f8c-b414-c15b29caea6e\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " Nov 02 18:11:47.198435 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.198399 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-cacerts\" (UniqueName: \"kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-cacerts\") pod \"10a632f9-0197-4f8c-b414-c15b29caea6e\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " Nov 02 18:11:47.198488 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.198435 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-csr-dns-cert\") pod \"10a632f9-0197-4f8c-b414-c15b29caea6e\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " Nov 02 18:11:47.198488 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.198476 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-kubeconfig\") pod \"10a632f9-0197-4f8c-b414-c15b29caea6e\" (UID: \"10a632f9-0197-4f8c-b414-c15b29caea6e\") " Nov 02 18:11:47.198715 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.198682 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-csr-ca-configmap" pod "10a632f9-0197-4f8c-b414-c15b29caea6e" (UID: "10a632f9-0197-4f8c-b414-c15b29caea6e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:47.199258 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.199228 4860 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-csr-ca-configmap\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:47.201265 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.200683 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-token" pod "10a632f9-0197-4f8c-b414-c15b29caea6e" (UID: "10a632f9-0197-4f8c-b414-c15b29caea6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:47.201265 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.200735 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-cacerts" pod "10a632f9-0197-4f8c-b414-c15b29caea6e" (UID: "10a632f9-0197-4f8c-b414-c15b29caea6e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:11:47.201265 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.201228 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a632f9-0197-4f8c-b414-c15b29caea6e-kube-api-access-whsl4" pod "10a632f9-0197-4f8c-b414-c15b29caea6e" (UID: "10a632f9-0197-4f8c-b414-c15b29caea6e"). InnerVolumeSpecName "kube-api-access-whsl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:47.201506 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.201481 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a632f9-0197-4f8c-b414-c15b29caea6e-local-certs" pod "10a632f9-0197-4f8c-b414-c15b29caea6e" (UID: "10a632f9-0197-4f8c-b414-c15b29caea6e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:47.202165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.202142 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-csr-dns-cert" pod "10a632f9-0197-4f8c-b414-c15b29caea6e" (UID: "10a632f9-0197-4f8c-b414-c15b29caea6e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:11:47.202400 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.202379 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-kubeconfig" pod "10a632f9-0197-4f8c-b414-c15b29caea6e" (UID: "10a632f9-0197-4f8c-b414-c15b29caea6e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:11:47.300288 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.300255 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:47.300288 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.300286 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whsl4\" (UniqueName: \"kubernetes.io/projected/10a632f9-0197-4f8c-b414-c15b29caea6e-kube-api-access-whsl4\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:47.300486 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.300296 4860 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a632f9-0197-4f8c-b414-c15b29caea6e-local-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:47.300486 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.300305 4860 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-cacerts\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:47.300486 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.300314 4860 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-csr-dns-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:47.300486 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.300321 4860 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10a632f9-0197-4f8c-b414-c15b29caea6e-istio-kubeconfig\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:47.506288 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.506206 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb27095-a4ff-429a-91be-6f3e28872f18" path="/var/lib/kubelet/pods/1fb27095-a4ff-429a-91be-6f3e28872f18/volumes" Nov 02 18:11:47.506701 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.506684 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" path="/var/lib/kubelet/pods/6975e1bd-ced6-4be9-8808-2e6fa6ff2a89/volumes" Nov 02 18:11:47.507168 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:47.507151 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9293d64-b25e-4076-b707-fc65717da6a8" path="/var/lib/kubelet/pods/e9293d64-b25e-4076-b707-fc65717da6a8/volumes" Nov 02 18:11:48.156695 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:48.156661 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" Nov 02 18:11:48.156695 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:48.156687 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5" event={"ID":"10a632f9-0197-4f8c-b414-c15b29caea6e","Type":"ContainerDied","Data":"c4d7e2519c4f03fecb7d6f58620fbb947b6da733f4bc269fe259e2b2d3d834cd"} Nov 02 18:11:48.157211 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:48.156745 4860 scope.go:122] "RemoveContainer" containerID="b2e39f38ce529a4f34acf22b28591824ea70d6c4ad384d6e00d423dfccb5cf5a" Nov 02 18:11:48.212274 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:48.212244 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5"] Nov 02 18:11:48.223587 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:48.223557 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9kkj5"] Nov 02 18:11:49.501420 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:49.501388 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10a632f9-0197-4f8c-b414-c15b29caea6e" path="/var/lib/kubelet/pods/10a632f9-0197-4f8c-b414-c15b29caea6e/volumes" Nov 02 18:11:51.835050 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:11:51.835005 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:11:52.166427 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.166390 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk"] Nov 02 18:11:52.203392 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.203360 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj"] Nov 02 18:11:52.207724 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.206805 4860 generic.go:356] "Generic (PLEG): container finished" podID="9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" containerID="9c789f0f324a1d3863d433a4bd09ad474e0df306700e59720602436ef13fc6ff" exitCode=0 Nov 02 18:11:52.207724 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.206945 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" event={"ID":"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9","Type":"ContainerDied","Data":"9c789f0f324a1d3863d433a4bd09ad474e0df306700e59720602436ef13fc6ff"} Nov 02 18:11:52.211894 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.210185 4860 generic.go:356] "Generic (PLEG): container finished" podID="b53fb7b2-d91c-409b-bf83-e24b95a7608f" containerID="30faeedd7646bee6bf8c2c08acf678ef95aee941ff9af52570e2eb64031430ad" exitCode=0 Nov 02 18:11:52.211894 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.210270 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" event={"ID":"b53fb7b2-d91c-409b-bf83-e24b95a7608f","Type":"ContainerDied","Data":"30faeedd7646bee6bf8c2c08acf678ef95aee941ff9af52570e2eb64031430ad"} Nov 02 18:11:52.213978 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.212711 4860 generic.go:356] "Generic (PLEG): container finished" podID="dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" containerID="ff641b2cf9a1dd6b53442dd633731a641f214cafc78ca1cd3fa26e8ac1693ae0" exitCode=0 Nov 02 18:11:52.214426 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.214208 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" event={"ID":"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3","Type":"ContainerDied","Data":"ff641b2cf9a1dd6b53442dd633731a641f214cafc78ca1cd3fa26e8ac1693ae0"} Nov 02 18:11:52.218699 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.218673 4860 generic.go:356] "Generic (PLEG): container finished" podID="c92d9c87-7ba0-4d6e-b659-9e916730c79e" containerID="71d866d15c5ebd732f0d33a1bda18a78921026df493328a1a5e9f5fab9f1a050" exitCode=0 Nov 02 18:11:52.218987 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.218844 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" event={"ID":"c92d9c87-7ba0-4d6e-b659-9e916730c79e","Type":"ContainerDied","Data":"71d866d15c5ebd732f0d33a1bda18a78921026df493328a1a5e9f5fab9f1a050"} Nov 02 18:11:52.222696 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.222669 4860 generic.go:356] "Generic (PLEG): container finished" podID="62b05251-7074-41bc-8994-c1051abee475" containerID="5a4a2440f0ea022d0c762cefdfcaa529ea8b44e559e079baffd8fd191a54c120" exitCode=0 Nov 02 18:11:52.222814 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.222724 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" event={"ID":"62b05251-7074-41bc-8994-c1051abee475","Type":"ContainerDied","Data":"5a4a2440f0ea022d0c762cefdfcaa529ea8b44e559e079baffd8fd191a54c120"} Nov 02 18:11:52.222814 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.222770 4860 scope.go:122] "RemoveContainer" containerID="afeda81a9e268826c31617880458f51bce447ac95759eea3a4e619b0cd6115a9" Nov 02 18:11:52.236185 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.236155 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 18:11:52.315548 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.315456 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-podinfo\") pod \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " Nov 02 18:11:52.315922 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.315897 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-workload-socket\") pod \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " Nov 02 18:11:52.316090 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.316072 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-credential-socket\") pod \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " Nov 02 18:11:52.316292 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.316278 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-token\" (UniqueName: \"kubernetes.io/projected/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-token\") pod \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " Nov 02 18:11:52.316444 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.316428 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-data\") pod \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " Nov 02 18:11:52.316593 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.316581 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-envoy\") pod \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " Nov 02 18:11:52.316729 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.316715 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-workload-certs\") pod \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " Nov 02 18:11:52.316902 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.316885 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istiod-ca-cert\") pod \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " Nov 02 18:11:52.317036 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.317021 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-kube-api-access-knghs\" (UniqueName: \"kubernetes.io/projected/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-kube-api-access-knghs\") pod \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\" (UID: \"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3\") " Nov 02 18:11:52.317832 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.317801 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-workload-socket" pod "dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" (UID: "dc8a063b-3299-4bb8-a4d8-1eb79e6adca3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.317934 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.317901 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istiod-ca-cert" pod "dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" (UID: "dc8a063b-3299-4bb8-a4d8-1eb79e6adca3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:52.318612 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.318174 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-credential-socket" pod "dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" (UID: "dc8a063b-3299-4bb8-a4d8-1eb79e6adca3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.318612 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.318217 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-workload-certs" pod "dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" (UID: "dc8a063b-3299-4bb8-a4d8-1eb79e6adca3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.318612 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.318567 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.318612 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.318589 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.319223 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.319203 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-data" pod "dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" (UID: "dc8a063b-3299-4bb8-a4d8-1eb79e6adca3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.323290 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.323256 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-token" pod "dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" (UID: "dc8a063b-3299-4bb8-a4d8-1eb79e6adca3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.323544 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.323518 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-kube-api-access-knghs" pod "dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" (UID: "dc8a063b-3299-4bb8-a4d8-1eb79e6adca3"). InnerVolumeSpecName "kube-api-access-knghs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.328136 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.328095 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-envoy" pod "dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" (UID: "dc8a063b-3299-4bb8-a4d8-1eb79e6adca3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.333288 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.333260 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-podinfo" pod "dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" (UID: "dc8a063b-3299-4bb8-a4d8-1eb79e6adca3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:52.373256 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.372930 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 18:11:52.423113 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.421493 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knghs\" (UniqueName: \"kubernetes.io/projected/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-kube-api-access-knghs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.423113 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.421526 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.423113 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.421541 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.423113 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.421554 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.423113 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.421568 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.423113 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.421580 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.423113 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.421592 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.447559 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.447231 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 18:11:52.456291 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.456265 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 18:11:52.468836 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.468808 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 18:11:52.523079 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523045 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-credential-socket\") pod \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " Nov 02 18:11:52.523268 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523105 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-istio-data\") pod \"62b05251-7074-41bc-8994-c1051abee475\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " Nov 02 18:11:52.523268 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523171 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-data\") pod \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " Nov 02 18:11:52.523268 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523200 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b05251-7074-41bc-8994-c1051abee475-kube-api-access-2jrsn\" (UniqueName: \"kubernetes.io/projected/62b05251-7074-41bc-8994-c1051abee475-kube-api-access-2jrsn\") pod \"62b05251-7074-41bc-8994-c1051abee475\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " Nov 02 18:11:52.523268 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523231 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-workload-socket\") pod \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " Nov 02 18:11:52.523268 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523256 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b05251-7074-41bc-8994-c1051abee475-istio-token\" (UniqueName: \"kubernetes.io/projected/62b05251-7074-41bc-8994-c1051abee475-istio-token\") pod \"62b05251-7074-41bc-8994-c1051abee475\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " Nov 02 18:11:52.523534 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523293 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istiod-ca-cert\") pod \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " Nov 02 18:11:52.523534 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523330 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-envoy\") pod \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " Nov 02 18:11:52.523534 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523364 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-workload-certs\") pod \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " Nov 02 18:11:52.523534 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523395 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-data\") pod \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " Nov 02 18:11:52.523534 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523389 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-credential-socket" pod "c92d9c87-7ba0-4d6e-b659-9e916730c79e" (UID: "c92d9c87-7ba0-4d6e-b659-9e916730c79e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.523534 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523441 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-token\" (UniqueName: \"kubernetes.io/projected/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-token\") pod \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " Nov 02 18:11:52.523534 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523449 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-istio-data" pod "62b05251-7074-41bc-8994-c1051abee475" (UID: "62b05251-7074-41bc-8994-c1051abee475"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.523534 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523485 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-workload-socket\") pod \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " Nov 02 18:11:52.523534 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523519 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c92d9c87-7ba0-4d6e-b659-9e916730c79e-kube-api-access-xnqdr\" (UniqueName: \"kubernetes.io/projected/c92d9c87-7ba0-4d6e-b659-9e916730c79e-kube-api-access-xnqdr\") pod \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " Nov 02 18:11:52.524066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523559 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-credential-socket\") pod \"62b05251-7074-41bc-8994-c1051abee475\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " Nov 02 18:11:52.524066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523595 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b53fb7b2-d91c-409b-bf83-e24b95a7608f-kube-api-access-mc2pw\" (UniqueName: \"kubernetes.io/projected/b53fb7b2-d91c-409b-bf83-e24b95a7608f-kube-api-access-mc2pw\") pod \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " Nov 02 18:11:52.524066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523631 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-credential-socket\") pod \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " Nov 02 18:11:52.524066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523649 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-workload-certs" pod "b53fb7b2-d91c-409b-bf83-e24b95a7608f" (UID: "b53fb7b2-d91c-409b-bf83-e24b95a7608f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.524066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523671 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istiod-ca-cert\") pod \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " Nov 02 18:11:52.524066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523725 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-kube-api-access-hpx6v\" (UniqueName: \"kubernetes.io/projected/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-kube-api-access-hpx6v\") pod \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " Nov 02 18:11:52.524066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523764 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-workload-certs\") pod \"62b05251-7074-41bc-8994-c1051abee475\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " Nov 02 18:11:52.524066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523801 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-workload-socket\") pod \"62b05251-7074-41bc-8994-c1051abee475\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " Nov 02 18:11:52.524066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523835 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62b05251-7074-41bc-8994-c1051abee475-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b05251-7074-41bc-8994-c1051abee475-istio-podinfo\") pod \"62b05251-7074-41bc-8994-c1051abee475\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " Nov 02 18:11:52.524066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523836 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-data" pod "b53fb7b2-d91c-409b-bf83-e24b95a7608f" (UID: "b53fb7b2-d91c-409b-bf83-e24b95a7608f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.524066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523891 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-workload-socket" pod "9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" (UID: "9e2cf636-17c0-4940-84d5-4e7f9faa7dc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.524066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.523877 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-token\" (UniqueName: \"kubernetes.io/projected/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-token\") pod \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " Nov 02 18:11:52.524066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.524003 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-workload-certs\") pod \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " Nov 02 18:11:52.524710 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.524099 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-token\" (UniqueName: \"kubernetes.io/projected/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-token\") pod \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " Nov 02 18:11:52.524710 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.524289 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-workload-socket\") pod \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " Nov 02 18:11:52.524710 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.524407 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62b05251-7074-41bc-8994-c1051abee475-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b05251-7074-41bc-8994-c1051abee475-istiod-ca-cert\") pod \"62b05251-7074-41bc-8994-c1051abee475\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " Nov 02 18:11:52.524710 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.524485 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-podinfo\") pod \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " Nov 02 18:11:52.524710 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.524517 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-envoy\") pod \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " Nov 02 18:11:52.524710 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.524542 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-workload-certs" pod "c92d9c87-7ba0-4d6e-b659-9e916730c79e" (UID: "c92d9c87-7ba0-4d6e-b659-9e916730c79e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.524710 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.524572 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-credential-socket\") pod \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " Nov 02 18:11:52.524710 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.524612 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istiod-ca-cert" pod "c92d9c87-7ba0-4d6e-b659-9e916730c79e" (UID: "c92d9c87-7ba0-4d6e-b659-9e916730c79e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:52.524710 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.524612 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-data\") pod \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " Nov 02 18:11:52.524710 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.524673 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-podinfo\") pod \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\" (UID: \"c92d9c87-7ba0-4d6e-b659-9e916730c79e\") " Nov 02 18:11:52.525256 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.524722 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istiod-ca-cert\") pod \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " Nov 02 18:11:52.525256 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.524764 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-istio-envoy\") pod \"62b05251-7074-41bc-8994-c1051abee475\" (UID: \"62b05251-7074-41bc-8994-c1051abee475\") " Nov 02 18:11:52.525256 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.524809 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-envoy\") pod \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " Nov 02 18:11:52.525256 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.524867 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-workload-socket" pod "b53fb7b2-d91c-409b-bf83-e24b95a7608f" (UID: "b53fb7b2-d91c-409b-bf83-e24b95a7608f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.525256 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.524982 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-podinfo\") pod \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\" (UID: \"b53fb7b2-d91c-409b-bf83-e24b95a7608f\") " Nov 02 18:11:52.525256 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.525064 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-workload-certs\") pod \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\" (UID: \"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9\") " Nov 02 18:11:52.525256 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.525071 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-data" pod "c92d9c87-7ba0-4d6e-b659-9e916730c79e" (UID: "c92d9c87-7ba0-4d6e-b659-9e916730c79e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.525256 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.525212 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-credential-socket" pod "b53fb7b2-d91c-409b-bf83-e24b95a7608f" (UID: "b53fb7b2-d91c-409b-bf83-e24b95a7608f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.525675 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.525506 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istiod-ca-cert" pod "b53fb7b2-d91c-409b-bf83-e24b95a7608f" (UID: "b53fb7b2-d91c-409b-bf83-e24b95a7608f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:52.526033 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.526010 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-workload-certs" pod "62b05251-7074-41bc-8994-c1051abee475" (UID: "62b05251-7074-41bc-8994-c1051abee475"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.526183 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.526087 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.526183 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.526110 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.526183 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.526126 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.526183 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.526140 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.526183 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.526156 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.526183 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.526172 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.526183 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.526185 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.526724 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.526198 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.526724 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.526212 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.526724 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.526228 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.526724 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.526243 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.526724 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.526136 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-workload-socket" pod "62b05251-7074-41bc-8994-c1051abee475" (UID: "62b05251-7074-41bc-8994-c1051abee475"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.526724 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.526242 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-workload-certs" pod "9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" (UID: "9e2cf636-17c0-4940-84d5-4e7f9faa7dc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.526724 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.526451 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-data" pod "9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" (UID: "9e2cf636-17c0-4940-84d5-4e7f9faa7dc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.526724 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.526707 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-workload-socket" pod "c92d9c87-7ba0-4d6e-b659-9e916730c79e" (UID: "c92d9c87-7ba0-4d6e-b659-9e916730c79e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.527702 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.527419 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-credential-socket" pod "9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" (UID: "9e2cf636-17c0-4940-84d5-4e7f9faa7dc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.528717 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.528436 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b05251-7074-41bc-8994-c1051abee475-istiod-ca-cert" pod "62b05251-7074-41bc-8994-c1051abee475" (UID: "62b05251-7074-41bc-8994-c1051abee475"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:52.529204 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.529166 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-kube-api-access-hpx6v" pod "9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" (UID: "9e2cf636-17c0-4940-84d5-4e7f9faa7dc9"). InnerVolumeSpecName "kube-api-access-hpx6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.530056 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.530030 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-credential-socket" pod "62b05251-7074-41bc-8994-c1051abee475" (UID: "62b05251-7074-41bc-8994-c1051abee475"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.530284 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.530243 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istiod-ca-cert" pod "9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" (UID: "9e2cf636-17c0-4940-84d5-4e7f9faa7dc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:52.531123 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.531088 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-envoy" pod "9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" (UID: "9e2cf636-17c0-4940-84d5-4e7f9faa7dc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.531342 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.531317 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92d9c87-7ba0-4d6e-b659-9e916730c79e-kube-api-access-xnqdr" pod "c92d9c87-7ba0-4d6e-b659-9e916730c79e" (UID: "c92d9c87-7ba0-4d6e-b659-9e916730c79e"). InnerVolumeSpecName "kube-api-access-xnqdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.531793 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.531772 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-podinfo" pod "b53fb7b2-d91c-409b-bf83-e24b95a7608f" (UID: "b53fb7b2-d91c-409b-bf83-e24b95a7608f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:52.532440 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.532422 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-podinfo" pod "9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" (UID: "9e2cf636-17c0-4940-84d5-4e7f9faa7dc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:52.533305 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.533282 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-envoy" pod "c92d9c87-7ba0-4d6e-b659-9e916730c79e" (UID: "c92d9c87-7ba0-4d6e-b659-9e916730c79e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.533402 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.533290 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62b05251-7074-41bc-8994-c1051abee475-istio-podinfo" pod "62b05251-7074-41bc-8994-c1051abee475" (UID: "62b05251-7074-41bc-8994-c1051abee475"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:52.534463 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.534433 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-token" pod "9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" (UID: "9e2cf636-17c0-4940-84d5-4e7f9faa7dc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.534463 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.534456 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b05251-7074-41bc-8994-c1051abee475-istio-token" pod "62b05251-7074-41bc-8994-c1051abee475" (UID: "62b05251-7074-41bc-8994-c1051abee475"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.534616 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.534464 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-token" pod "c92d9c87-7ba0-4d6e-b659-9e916730c79e" (UID: "c92d9c87-7ba0-4d6e-b659-9e916730c79e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.534616 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.534440 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-envoy" pod "b53fb7b2-d91c-409b-bf83-e24b95a7608f" (UID: "b53fb7b2-d91c-409b-bf83-e24b95a7608f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.534737 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.534714 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-istio-envoy" pod "62b05251-7074-41bc-8994-c1051abee475" (UID: "62b05251-7074-41bc-8994-c1051abee475"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:52.535253 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.535233 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-podinfo" pod "c92d9c87-7ba0-4d6e-b659-9e916730c79e" (UID: "c92d9c87-7ba0-4d6e-b659-9e916730c79e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:52.535328 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.535306 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b53fb7b2-d91c-409b-bf83-e24b95a7608f-kube-api-access-mc2pw" pod "b53fb7b2-d91c-409b-bf83-e24b95a7608f" (UID: "b53fb7b2-d91c-409b-bf83-e24b95a7608f"). InnerVolumeSpecName "kube-api-access-mc2pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.535696 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.535678 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b05251-7074-41bc-8994-c1051abee475-kube-api-access-2jrsn" pod "62b05251-7074-41bc-8994-c1051abee475" (UID: "62b05251-7074-41bc-8994-c1051abee475"). InnerVolumeSpecName "kube-api-access-2jrsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.536182 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.536160 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-token" pod "b53fb7b2-d91c-409b-bf83-e24b95a7608f" (UID: "b53fb7b2-d91c-409b-bf83-e24b95a7608f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:52.627700 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627611 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.627700 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627644 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc2pw\" (UniqueName: \"kubernetes.io/projected/b53fb7b2-d91c-409b-bf83-e24b95a7608f-kube-api-access-mc2pw\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.627700 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627658 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpx6v\" (UniqueName: \"kubernetes.io/projected/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-kube-api-access-hpx6v\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.627700 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627672 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.627700 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627685 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.627700 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627697 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b05251-7074-41bc-8994-c1051abee475-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.628137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627709 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.628137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627723 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.628137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627734 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.628137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627746 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b05251-7074-41bc-8994-c1051abee475-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.628137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627758 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.628137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627771 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.628137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627783 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.628137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627794 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.628137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627806 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.628137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627818 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.628137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627830 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b05251-7074-41bc-8994-c1051abee475-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.628137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627841 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.628137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627891 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.628137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627904 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.628137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627918 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jrsn\" (UniqueName: \"kubernetes.io/projected/62b05251-7074-41bc-8994-c1051abee475-kube-api-access-2jrsn\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.628137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627930 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b05251-7074-41bc-8994-c1051abee475-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.628137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627944 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b53fb7b2-d91c-409b-bf83-e24b95a7608f-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.628137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627956 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c92d9c87-7ba0-4d6e-b659-9e916730c79e-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.628137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.627970 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnqdr\" (UniqueName: \"kubernetes.io/projected/c92d9c87-7ba0-4d6e-b659-9e916730c79e-kube-api-access-xnqdr\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:52.897528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.897490 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:11:52.985659 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.985624 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw"] Nov 02 18:11:52.986099 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.986064 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" podUID="c12481ab-3025-40e3-a9a7-50b44df24d70" containerName="istio-proxy" containerID="cri-o://0431ac7eb19a2ef02ca11ea864d1429253ec222e364a4510af75c05cf73bc61d" gracePeriod=30 Nov 02 18:11:52.992179 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.992142 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4"] Nov 02 18:11:52.992546 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:52.992521 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" podUID="a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" containerName="istio-proxy" containerID="cri-o://9b5f0c1e197f557ea4d9e9791cf957de49ace2a5cc7fb7bbcc0fcf335c7f0b64" gracePeriod=30 Nov 02 18:11:53.034401 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.034365 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istiod-ca-cert\") pod \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " Nov 02 18:11:53.034644 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.034611 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-workload-certs\") pod \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " Nov 02 18:11:53.034796 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.034729 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-kube-api-access-dzvqk\" (UniqueName: \"kubernetes.io/projected/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-kube-api-access-dzvqk\") pod \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " Nov 02 18:11:53.034923 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.034833 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-workload-certs" pod "e31cdb01-9f71-4eae-ad26-6f1e19915ffd" (UID: "e31cdb01-9f71-4eae-ad26-6f1e19915ffd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:53.034923 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.034889 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-data\") pod \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " Nov 02 18:11:53.035066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.034979 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-workload-socket\") pod \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " Nov 02 18:11:53.035066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.035024 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-podinfo\") pod \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " Nov 02 18:11:53.035186 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.035130 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-data" pod "e31cdb01-9f71-4eae-ad26-6f1e19915ffd" (UID: "e31cdb01-9f71-4eae-ad26-6f1e19915ffd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:53.035258 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.035236 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istiod-ca-cert" pod "e31cdb01-9f71-4eae-ad26-6f1e19915ffd" (UID: "e31cdb01-9f71-4eae-ad26-6f1e19915ffd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:53.036941 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.035582 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-token\" (UniqueName: \"kubernetes.io/projected/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-token\") pod \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " Nov 02 18:11:53.036941 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.035622 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-credential-socket\") pod \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " Nov 02 18:11:53.036941 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.035674 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-envoy\") pod \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\" (UID: \"e31cdb01-9f71-4eae-ad26-6f1e19915ffd\") " Nov 02 18:11:53.036941 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.035904 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-workload-socket" pod "e31cdb01-9f71-4eae-ad26-6f1e19915ffd" (UID: "e31cdb01-9f71-4eae-ad26-6f1e19915ffd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:53.036941 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.036631 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:53.036941 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.036652 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:53.036941 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.036666 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:53.036941 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.036681 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:53.040062 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.037785 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-kube-api-access-dzvqk" pod "e31cdb01-9f71-4eae-ad26-6f1e19915ffd" (UID: "e31cdb01-9f71-4eae-ad26-6f1e19915ffd"). InnerVolumeSpecName "kube-api-access-dzvqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:53.040062 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.038001 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-credential-socket" pod "e31cdb01-9f71-4eae-ad26-6f1e19915ffd" (UID: "e31cdb01-9f71-4eae-ad26-6f1e19915ffd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:53.040062 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.039992 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-envoy" pod "e31cdb01-9f71-4eae-ad26-6f1e19915ffd" (UID: "e31cdb01-9f71-4eae-ad26-6f1e19915ffd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:53.040564 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.040524 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-podinfo" pod "e31cdb01-9f71-4eae-ad26-6f1e19915ffd" (UID: "e31cdb01-9f71-4eae-ad26-6f1e19915ffd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:53.042118 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.042093 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-token" pod "e31cdb01-9f71-4eae-ad26-6f1e19915ffd" (UID: "e31cdb01-9f71-4eae-ad26-6f1e19915ffd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:53.138063 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.137964 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:53.138328 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.138288 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:53.138504 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.138487 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:53.138632 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.138616 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:53.138753 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.138740 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzvqk\" (UniqueName: \"kubernetes.io/projected/e31cdb01-9f71-4eae-ad26-6f1e19915ffd-kube-api-access-dzvqk\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:53.247828 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.247787 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" event={"ID":"9e2cf636-17c0-4940-84d5-4e7f9faa7dc9","Type":"ContainerDied","Data":"9577d38f372cc2904e2adb171b9ce9cb5fb90bf66d632c03e6eecf4c127aa254"} Nov 02 18:11:53.248075 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.247895 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz" Nov 02 18:11:53.248075 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.247949 4860 scope.go:122] "RemoveContainer" containerID="9c789f0f324a1d3863d433a4bd09ad474e0df306700e59720602436ef13fc6ff" Nov 02 18:11:53.250125 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.250096 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" event={"ID":"b53fb7b2-d91c-409b-bf83-e24b95a7608f","Type":"ContainerDied","Data":"b4e40b546ae45993bb243ebce589b80d14a8208e0bb0ea298c1e0d01f9fc0ae4"} Nov 02 18:11:53.250289 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.250135 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf" Nov 02 18:11:53.252010 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.251982 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" Nov 02 18:11:53.252122 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.251983 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm" event={"ID":"dc8a063b-3299-4bb8-a4d8-1eb79e6adca3","Type":"ContainerDied","Data":"f865b73142b77b34c7a50fdabc7b3daff722014e2bdeb95b49c5218732cf8d04"} Nov 02 18:11:53.254069 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.254047 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" Nov 02 18:11:53.254069 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.254055 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh" event={"ID":"c92d9c87-7ba0-4d6e-b659-9e916730c79e","Type":"ContainerDied","Data":"ec5f0a01cb754f983839856f989c58efba71b821216c7ab113c7c472df504185"} Nov 02 18:11:53.260701 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.260675 4860 generic.go:356] "Generic (PLEG): container finished" podID="e31cdb01-9f71-4eae-ad26-6f1e19915ffd" containerID="9da9acd413ede11a467a5cf900fff9debafc3b5dce5118d9af0863fbd69f080d" exitCode=0 Nov 02 18:11:53.261082 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.260921 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" Nov 02 18:11:53.261272 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.260938 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" event={"ID":"e31cdb01-9f71-4eae-ad26-6f1e19915ffd","Type":"ContainerDied","Data":"9da9acd413ede11a467a5cf900fff9debafc3b5dce5118d9af0863fbd69f080d"} Nov 02 18:11:53.261415 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.261396 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb" event={"ID":"e31cdb01-9f71-4eae-ad26-6f1e19915ffd","Type":"ContainerDied","Data":"ea9cf1ca6a488e802fa772efbc4eca2cb21a38b22ae36911f3f81706cfa45267"} Nov 02 18:11:53.263822 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.263800 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" event={"ID":"62b05251-7074-41bc-8994-c1051abee475","Type":"ContainerDied","Data":"a176ffc3ede0c1545a4eb59f8a5abc48b18bf2cffce20473c05058117126a545"} Nov 02 18:11:53.264150 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.263981 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v" Nov 02 18:11:53.264431 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.264415 4860 scope.go:122] "RemoveContainer" containerID="30faeedd7646bee6bf8c2c08acf678ef95aee941ff9af52570e2eb64031430ad" Nov 02 18:11:53.277683 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.277662 4860 scope.go:122] "RemoveContainer" containerID="ff641b2cf9a1dd6b53442dd633731a641f214cafc78ca1cd3fa26e8ac1693ae0" Nov 02 18:11:53.294218 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.294193 4860 scope.go:122] "RemoveContainer" containerID="71d866d15c5ebd732f0d33a1bda18a78921026df493328a1a5e9f5fab9f1a050" Nov 02 18:11:53.308243 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.308216 4860 scope.go:122] "RemoveContainer" containerID="9da9acd413ede11a467a5cf900fff9debafc3b5dce5118d9af0863fbd69f080d" Nov 02 18:11:53.322305 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.322264 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz"] Nov 02 18:11:53.330980 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.330953 4860 scope.go:122] "RemoveContainer" containerID="9da9acd413ede11a467a5cf900fff9debafc3b5dce5118d9af0863fbd69f080d" Nov 02 18:11:53.331340 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:11:53.331310 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9da9acd413ede11a467a5cf900fff9debafc3b5dce5118d9af0863fbd69f080d\": container with ID starting with 9da9acd413ede11a467a5cf900fff9debafc3b5dce5118d9af0863fbd69f080d not found: ID does not exist" containerID="9da9acd413ede11a467a5cf900fff9debafc3b5dce5118d9af0863fbd69f080d" Nov 02 18:11:53.331505 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.331369 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9da9acd413ede11a467a5cf900fff9debafc3b5dce5118d9af0863fbd69f080d"} err="failed to get container status \"9da9acd413ede11a467a5cf900fff9debafc3b5dce5118d9af0863fbd69f080d\": rpc error: code = NotFound desc = could not find container \"9da9acd413ede11a467a5cf900fff9debafc3b5dce5118d9af0863fbd69f080d\": container with ID starting with 9da9acd413ede11a467a5cf900fff9debafc3b5dce5118d9af0863fbd69f080d not found: ID does not exist" Nov 02 18:11:53.331505 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.331393 4860 scope.go:122] "RemoveContainer" containerID="5a4a2440f0ea022d0c762cefdfcaa529ea8b44e559e079baffd8fd191a54c120" Nov 02 18:11:53.332619 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.332596 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-fmpfz"] Nov 02 18:11:53.437051 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.436956 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb"] Nov 02 18:11:53.444234 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.444208 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8tbkb"] Nov 02 18:11:53.477122 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.477098 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh"] Nov 02 18:11:53.504721 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.504692 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" path="/var/lib/kubelet/pods/9e2cf636-17c0-4940-84d5-4e7f9faa7dc9/volumes" Nov 02 18:11:53.505270 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.505250 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e31cdb01-9f71-4eae-ad26-6f1e19915ffd" path="/var/lib/kubelet/pods/e31cdb01-9f71-4eae-ad26-6f1e19915ffd/volumes" Nov 02 18:11:53.505607 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.505591 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-b82fh"] Nov 02 18:11:53.533290 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.533251 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf"] Nov 02 18:11:53.539605 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.539568 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-94hmf"] Nov 02 18:11:53.575537 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.575488 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v"] Nov 02 18:11:53.579939 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.579914 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-v9l5v"] Nov 02 18:11:53.613958 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.613930 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm"] Nov 02 18:11:53.617938 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:53.617902 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-sxmcm"] Nov 02 18:11:55.504176 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:55.501993 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62b05251-7074-41bc-8994-c1051abee475" path="/var/lib/kubelet/pods/62b05251-7074-41bc-8994-c1051abee475/volumes" Nov 02 18:11:55.504176 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:55.502898 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b53fb7b2-d91c-409b-bf83-e24b95a7608f" path="/var/lib/kubelet/pods/b53fb7b2-d91c-409b-bf83-e24b95a7608f/volumes" Nov 02 18:11:55.505335 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:55.503680 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c92d9c87-7ba0-4d6e-b659-9e916730c79e" path="/var/lib/kubelet/pods/c92d9c87-7ba0-4d6e-b659-9e916730c79e/volumes" Nov 02 18:11:55.509056 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:55.506106 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" path="/var/lib/kubelet/pods/dc8a063b-3299-4bb8-a4d8-1eb79e6adca3/volumes" Nov 02 18:11:55.540696 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:55.540661 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd"] Nov 02 18:11:55.540954 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:55.540923 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" podUID="d4332a04-bc02-4fce-925b-df037762c76f" containerName="maas-api" containerID="cri-o://e447a6570860e5b8a68b41bebf8f01847ac5d47119a361fe75ca14bdbdb62072" gracePeriod=30 Nov 02 18:11:55.889273 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:55.889244 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" Nov 02 18:11:55.992399 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:55.992358 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4332a04-bc02-4fce-925b-df037762c76f-kube-api-access-9t2nr\" (UniqueName: \"kubernetes.io/projected/d4332a04-bc02-4fce-925b-df037762c76f-kube-api-access-9t2nr\") pod \"d4332a04-bc02-4fce-925b-df037762c76f\" (UID: \"d4332a04-bc02-4fce-925b-df037762c76f\") " Nov 02 18:11:55.992602 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:55.992525 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4332a04-bc02-4fce-925b-df037762c76f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4332a04-bc02-4fce-925b-df037762c76f-metrics-certs\") pod \"d4332a04-bc02-4fce-925b-df037762c76f\" (UID: \"d4332a04-bc02-4fce-925b-df037762c76f\") " Nov 02 18:11:55.992688 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:55.992638 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4332a04-bc02-4fce-925b-df037762c76f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4332a04-bc02-4fce-925b-df037762c76f-maas-api-tls\") pod \"d4332a04-bc02-4fce-925b-df037762c76f\" (UID: \"d4332a04-bc02-4fce-925b-df037762c76f\") " Nov 02 18:11:55.992757 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:55.992744 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4332a04-bc02-4fce-925b-df037762c76f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4332a04-bc02-4fce-925b-df037762c76f-openshift-service-ca\") pod \"d4332a04-bc02-4fce-925b-df037762c76f\" (UID: \"d4332a04-bc02-4fce-925b-df037762c76f\") " Nov 02 18:11:55.993751 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:55.993721 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4332a04-bc02-4fce-925b-df037762c76f-openshift-service-ca" pod "d4332a04-bc02-4fce-925b-df037762c76f" (UID: "d4332a04-bc02-4fce-925b-df037762c76f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:55.996424 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:55.996397 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4332a04-bc02-4fce-925b-df037762c76f-kube-api-access-9t2nr" pod "d4332a04-bc02-4fce-925b-df037762c76f" (UID: "d4332a04-bc02-4fce-925b-df037762c76f"). InnerVolumeSpecName "kube-api-access-9t2nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:55.996424 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:55.996410 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4332a04-bc02-4fce-925b-df037762c76f-maas-api-tls" pod "d4332a04-bc02-4fce-925b-df037762c76f" (UID: "d4332a04-bc02-4fce-925b-df037762c76f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:11:55.996757 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:55.996739 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4332a04-bc02-4fce-925b-df037762c76f-metrics-certs" pod "d4332a04-bc02-4fce-925b-df037762c76f" (UID: "d4332a04-bc02-4fce-925b-df037762c76f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:11:56.095289 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:56.095191 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4332a04-bc02-4fce-925b-df037762c76f-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:56.095289 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:56.095232 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4332a04-bc02-4fce-925b-df037762c76f-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:56.095289 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:56.095246 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4332a04-bc02-4fce-925b-df037762c76f-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:56.095289 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:56.095260 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t2nr\" (UniqueName: \"kubernetes.io/projected/d4332a04-bc02-4fce-925b-df037762c76f-kube-api-access-9t2nr\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:56.296424 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:56.296385 4860 generic.go:356] "Generic (PLEG): container finished" podID="d4332a04-bc02-4fce-925b-df037762c76f" containerID="e447a6570860e5b8a68b41bebf8f01847ac5d47119a361fe75ca14bdbdb62072" exitCode=0 Nov 02 18:11:56.296595 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:56.296441 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" Nov 02 18:11:56.296595 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:56.296473 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" event={"ID":"d4332a04-bc02-4fce-925b-df037762c76f","Type":"ContainerDied","Data":"e447a6570860e5b8a68b41bebf8f01847ac5d47119a361fe75ca14bdbdb62072"} Nov 02 18:11:56.296595 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:56.296529 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd" event={"ID":"d4332a04-bc02-4fce-925b-df037762c76f","Type":"ContainerDied","Data":"a75c1c11ab902d0efd3647dadc38d0d9026e45d508160a68748a2da612963d8b"} Nov 02 18:11:56.296595 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:56.296560 4860 scope.go:122] "RemoveContainer" containerID="e447a6570860e5b8a68b41bebf8f01847ac5d47119a361fe75ca14bdbdb62072" Nov 02 18:11:56.309660 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:56.309639 4860 scope.go:122] "RemoveContainer" containerID="e447a6570860e5b8a68b41bebf8f01847ac5d47119a361fe75ca14bdbdb62072" Nov 02 18:11:56.309956 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:11:56.309936 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e447a6570860e5b8a68b41bebf8f01847ac5d47119a361fe75ca14bdbdb62072\": container with ID starting with e447a6570860e5b8a68b41bebf8f01847ac5d47119a361fe75ca14bdbdb62072 not found: ID does not exist" containerID="e447a6570860e5b8a68b41bebf8f01847ac5d47119a361fe75ca14bdbdb62072" Nov 02 18:11:56.310040 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:56.309964 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e447a6570860e5b8a68b41bebf8f01847ac5d47119a361fe75ca14bdbdb62072"} err="failed to get container status \"e447a6570860e5b8a68b41bebf8f01847ac5d47119a361fe75ca14bdbdb62072\": rpc error: code = NotFound desc = could not find container \"e447a6570860e5b8a68b41bebf8f01847ac5d47119a361fe75ca14bdbdb62072\": container with ID starting with e447a6570860e5b8a68b41bebf8f01847ac5d47119a361fe75ca14bdbdb62072 not found: ID does not exist" Nov 02 18:11:56.356838 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:56.356757 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd"] Nov 02 18:11:56.368789 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:56.368763 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8a209f-7f74587fcc-rf6wd"] Nov 02 18:11:57.167364 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:57.167321 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" podUID="bafb2593-6516-446c-b113-37bff6320729" containerName="payload-processing" containerID="cri-o://d0ab35b8d1c48779542d7bf2aed4481f66d9eb405a3b018669c98a3f2902b337" gracePeriod=25 Nov 02 18:11:57.204716 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:57.204664 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" podUID="75e58f61-43a2-4600-a380-549625283343" containerName="payload-pre-processing" containerID="cri-o://55f7c5a5ab471fc60cd7b430c1d6b6dc35a1ff33131fa7367f753f19d1b10179" gracePeriod=25 Nov 02 18:11:57.291481 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:57.291438 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" podUID="75e58f61-43a2-4600-a380-549625283343" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:11:57.291691 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:57.291495 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" podUID="bafb2593-6516-446c-b113-37bff6320729" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:11:57.501264 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:57.501183 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4332a04-bc02-4fce-925b-df037762c76f" path="/var/lib/kubelet/pods/d4332a04-bc02-4fce-925b-df037762c76f/volumes" Nov 02 18:11:58.327939 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.327821 4860 generic.go:356] "Generic (PLEG): container finished" podID="a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" containerID="9b5f0c1e197f557ea4d9e9791cf957de49ace2a5cc7fb7bbcc0fcf335c7f0b64" exitCode=0 Nov 02 18:11:58.328466 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.328054 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" event={"ID":"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96","Type":"ContainerDied","Data":"9b5f0c1e197f557ea4d9e9791cf957de49ace2a5cc7fb7bbcc0fcf335c7f0b64"} Nov 02 18:11:58.364705 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.355816 4860 generic.go:356] "Generic (PLEG): container finished" podID="c12481ab-3025-40e3-a9a7-50b44df24d70" containerID="0431ac7eb19a2ef02ca11ea864d1429253ec222e364a4510af75c05cf73bc61d" exitCode=0 Nov 02 18:11:58.364705 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.356061 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" event={"ID":"c12481ab-3025-40e3-a9a7-50b44df24d70","Type":"ContainerDied","Data":"0431ac7eb19a2ef02ca11ea864d1429253ec222e364a4510af75c05cf73bc61d"} Nov 02 18:11:58.413732 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.413700 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:11:58.438347 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.438324 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:11:58.512562 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.512523 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg"] Nov 02 18:11:58.534938 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.534906 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6"] Nov 02 18:11:58.539961 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.539937 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-token\" (UniqueName: \"kubernetes.io/projected/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-token\") pod \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " Nov 02 18:11:58.540220 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.540200 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istiod-ca-cert\") pod \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " Nov 02 18:11:58.540371 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.540355 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c12481ab-3025-40e3-a9a7-50b44df24d70-kube-api-access-b9w9g\" (UniqueName: \"kubernetes.io/projected/c12481ab-3025-40e3-a9a7-50b44df24d70-kube-api-access-b9w9g\") pod \"c12481ab-3025-40e3-a9a7-50b44df24d70\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " Nov 02 18:11:58.540516 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.540496 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-workload-certs\") pod \"c12481ab-3025-40e3-a9a7-50b44df24d70\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " Nov 02 18:11:58.540642 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.540628 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-credential-socket\") pod \"c12481ab-3025-40e3-a9a7-50b44df24d70\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " Nov 02 18:11:58.542218 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.542183 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-token" pod "a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" (UID: "a67e8cb6-152f-4cdd-a814-d1cfdbf99d96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:58.542331 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.542286 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-data\") pod \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " Nov 02 18:11:58.542392 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.542340 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-credential-socket\") pod \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " Nov 02 18:11:58.542392 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.542385 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-podinfo\") pod \"c12481ab-3025-40e3-a9a7-50b44df24d70\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " Nov 02 18:11:58.542662 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.542642 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-credential-socket" pod "c12481ab-3025-40e3-a9a7-50b44df24d70" (UID: "c12481ab-3025-40e3-a9a7-50b44df24d70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:58.542746 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.542590 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istiod-ca-cert" pod "a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" (UID: "a67e8cb6-152f-4cdd-a814-d1cfdbf99d96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:58.542905 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.542872 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-workload-socket\") pod \"c12481ab-3025-40e3-a9a7-50b44df24d70\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " Nov 02 18:11:58.542905 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.542879 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-data" pod "a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" (UID: "a67e8cb6-152f-4cdd-a814-d1cfdbf99d96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:58.543048 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.542927 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-kube-api-access-ncmvp\" (UniqueName: \"kubernetes.io/projected/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-kube-api-access-ncmvp\") pod \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " Nov 02 18:11:58.543048 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.542925 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-credential-socket" pod "a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" (UID: "a67e8cb6-152f-4cdd-a814-d1cfdbf99d96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:58.543048 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.542967 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-workload-socket\") pod \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " Nov 02 18:11:58.543233 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.543053 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-token\" (UniqueName: \"kubernetes.io/projected/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-token\") pod \"c12481ab-3025-40e3-a9a7-50b44df24d70\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " Nov 02 18:11:58.543233 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.543154 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-envoy\") pod \"c12481ab-3025-40e3-a9a7-50b44df24d70\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " Nov 02 18:11:58.543233 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.543171 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-workload-certs" pod "c12481ab-3025-40e3-a9a7-50b44df24d70" (UID: "c12481ab-3025-40e3-a9a7-50b44df24d70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:58.543233 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.543194 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-envoy\") pod \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " Nov 02 18:11:58.543487 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.543305 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-data\") pod \"c12481ab-3025-40e3-a9a7-50b44df24d70\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " Nov 02 18:11:58.543487 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.543341 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-workload-certs\") pod \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " Nov 02 18:11:58.543487 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.543369 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-workload-socket" pod "a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" (UID: "a67e8cb6-152f-4cdd-a814-d1cfdbf99d96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:58.543487 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.543413 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-podinfo\") pod \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\" (UID: \"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96\") " Nov 02 18:11:58.543487 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.543475 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c12481ab-3025-40e3-a9a7-50b44df24d70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c12481ab-3025-40e3-a9a7-50b44df24d70-istiod-ca-cert\") pod \"c12481ab-3025-40e3-a9a7-50b44df24d70\" (UID: \"c12481ab-3025-40e3-a9a7-50b44df24d70\") " Nov 02 18:11:58.543751 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.543720 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12481ab-3025-40e3-a9a7-50b44df24d70-kube-api-access-b9w9g" pod "c12481ab-3025-40e3-a9a7-50b44df24d70" (UID: "c12481ab-3025-40e3-a9a7-50b44df24d70"). InnerVolumeSpecName "kube-api-access-b9w9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:58.543945 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.543923 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-workload-socket" pod "c12481ab-3025-40e3-a9a7-50b44df24d70" (UID: "c12481ab-3025-40e3-a9a7-50b44df24d70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:58.544343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.544320 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.544493 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.544344 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.544493 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.544358 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.544493 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.544372 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9w9g\" (UniqueName: \"kubernetes.io/projected/c12481ab-3025-40e3-a9a7-50b44df24d70-kube-api-access-b9w9g\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.544493 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.544384 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.544493 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.544397 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.544493 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.544410 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.544493 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.544423 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.544493 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.544458 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.544493 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.543953 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-data" pod "c12481ab-3025-40e3-a9a7-50b44df24d70" (UID: "c12481ab-3025-40e3-a9a7-50b44df24d70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:58.545128 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.544497 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-workload-certs" pod "a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" (UID: "a67e8cb6-152f-4cdd-a814-d1cfdbf99d96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:58.545679 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.545659 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-kube-api-access-ncmvp" pod "a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" (UID: "a67e8cb6-152f-4cdd-a814-d1cfdbf99d96"). InnerVolumeSpecName "kube-api-access-ncmvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:58.545905 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.545840 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-podinfo" pod "c12481ab-3025-40e3-a9a7-50b44df24d70" (UID: "c12481ab-3025-40e3-a9a7-50b44df24d70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:58.547102 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.547066 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-token" pod "c12481ab-3025-40e3-a9a7-50b44df24d70" (UID: "c12481ab-3025-40e3-a9a7-50b44df24d70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:11:58.547191 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.547137 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12481ab-3025-40e3-a9a7-50b44df24d70-istiod-ca-cert" pod "c12481ab-3025-40e3-a9a7-50b44df24d70" (UID: "c12481ab-3025-40e3-a9a7-50b44df24d70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:11:58.547326 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.547302 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-podinfo" pod "a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" (UID: "a67e8cb6-152f-4cdd-a814-d1cfdbf99d96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:11:58.548546 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.548523 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-envoy" pod "c12481ab-3025-40e3-a9a7-50b44df24d70" (UID: "c12481ab-3025-40e3-a9a7-50b44df24d70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:58.548741 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.548720 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-envoy" pod "a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" (UID: "a67e8cb6-152f-4cdd-a814-d1cfdbf99d96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:11:58.645573 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.645481 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.645573 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.645513 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.645573 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.645526 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.645573 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.645537 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c12481ab-3025-40e3-a9a7-50b44df24d70-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.645573 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.645552 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.645573 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.645566 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncmvp\" (UniqueName: \"kubernetes.io/projected/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-kube-api-access-ncmvp\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.645573 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.645578 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.645954 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.645590 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c12481ab-3025-40e3-a9a7-50b44df24d70-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:58.645954 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:58.645603 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:11:59.371986 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:59.371946 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" Nov 02 18:11:59.372632 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:59.371985 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4" event={"ID":"a67e8cb6-152f-4cdd-a814-d1cfdbf99d96","Type":"ContainerDied","Data":"5209dd3ae5514a823ed41826ba08565f19f4b17eaba3c28adeab4333e6efbc78"} Nov 02 18:11:59.372632 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:59.372063 4860 scope.go:122] "RemoveContainer" containerID="9b5f0c1e197f557ea4d9e9791cf957de49ace2a5cc7fb7bbcc0fcf335c7f0b64" Nov 02 18:11:59.374258 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:59.373811 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" Nov 02 18:11:59.374258 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:59.373838 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw" event={"ID":"c12481ab-3025-40e3-a9a7-50b44df24d70","Type":"ContainerDied","Data":"9852168cfd0f04238df8505eec3d064e0cecd9ef6ec15f2c31977ec42f9e982b"} Nov 02 18:11:59.397834 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:59.397806 4860 scope.go:122] "RemoveContainer" containerID="0431ac7eb19a2ef02ca11ea864d1429253ec222e364a4510af75c05cf73bc61d" Nov 02 18:11:59.434268 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:59.434234 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw"] Nov 02 18:11:59.446504 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:59.446472 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-t66bw"] Nov 02 18:11:59.471141 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:59.471108 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4"] Nov 02 18:11:59.477607 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:59.477580 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-8xhr4"] Nov 02 18:11:59.504728 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:59.504698 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" path="/var/lib/kubelet/pods/a67e8cb6-152f-4cdd-a814-d1cfdbf99d96/volumes" Nov 02 18:11:59.505425 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:11:59.505403 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c12481ab-3025-40e3-a9a7-50b44df24d70" path="/var/lib/kubelet/pods/c12481ab-3025-40e3-a9a7-50b44df24d70/volumes" Nov 02 18:12:04.454796 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:04.454759 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk"] Nov 02 18:12:04.455333 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:04.455033 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" podUID="66504ddb-d869-4549-b870-61b42e58e871" containerName="maas-api" containerID="cri-o://060ba0b0295fab8763442d2f97fd2a58e32cf4e65d63f7f9b59611601e1cc01c" gracePeriod=30 Nov 02 18:12:04.814210 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:04.814183 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" Nov 02 18:12:04.916575 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:04.916539 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5"] Nov 02 18:12:04.916953 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:04.916793 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" podUID="719f739a-813b-4f51-9bd4-e523bba953af" containerName="maas-api" containerID="cri-o://6058ad0ba329ef2d321c271101c9de9309f3590be6840a815ba5d495c73ca212" gracePeriod=30 Nov 02 18:12:04.935211 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:04.935175 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66504ddb-d869-4549-b870-61b42e58e871-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66504ddb-d869-4549-b870-61b42e58e871-maas-api-tls\") pod \"66504ddb-d869-4549-b870-61b42e58e871\" (UID: \"66504ddb-d869-4549-b870-61b42e58e871\") " Nov 02 18:12:04.935584 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:04.935558 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66504ddb-d869-4549-b870-61b42e58e871-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66504ddb-d869-4549-b870-61b42e58e871-openshift-service-ca\") pod \"66504ddb-d869-4549-b870-61b42e58e871\" (UID: \"66504ddb-d869-4549-b870-61b42e58e871\") " Nov 02 18:12:04.935689 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:04.935606 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66504ddb-d869-4549-b870-61b42e58e871-kube-api-access-k69d4\" (UniqueName: \"kubernetes.io/projected/66504ddb-d869-4549-b870-61b42e58e871-kube-api-access-k69d4\") pod \"66504ddb-d869-4549-b870-61b42e58e871\" (UID: \"66504ddb-d869-4549-b870-61b42e58e871\") " Nov 02 18:12:04.935689 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:04.935641 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66504ddb-d869-4549-b870-61b42e58e871-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66504ddb-d869-4549-b870-61b42e58e871-metrics-certs\") pod \"66504ddb-d869-4549-b870-61b42e58e871\" (UID: \"66504ddb-d869-4549-b870-61b42e58e871\") " Nov 02 18:12:04.935966 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:04.935939 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66504ddb-d869-4549-b870-61b42e58e871-openshift-service-ca" pod "66504ddb-d869-4549-b870-61b42e58e871" (UID: "66504ddb-d869-4549-b870-61b42e58e871"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:04.936242 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:04.936221 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66504ddb-d869-4549-b870-61b42e58e871-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:04.943106 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:04.943072 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66504ddb-d869-4549-b870-61b42e58e871-metrics-certs" pod "66504ddb-d869-4549-b870-61b42e58e871" (UID: "66504ddb-d869-4549-b870-61b42e58e871"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:04.952188 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:04.952157 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66504ddb-d869-4549-b870-61b42e58e871-kube-api-access-k69d4" pod "66504ddb-d869-4549-b870-61b42e58e871" (UID: "66504ddb-d869-4549-b870-61b42e58e871"). InnerVolumeSpecName "kube-api-access-k69d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:04.965045 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:04.965012 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66504ddb-d869-4549-b870-61b42e58e871-maas-api-tls" pod "66504ddb-d869-4549-b870-61b42e58e871" (UID: "66504ddb-d869-4549-b870-61b42e58e871"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:04.998147 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:12:04.998116 4860 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719f739a_813b_4f51_9bd4_e523bba953af.slice/crio-6058ad0ba329ef2d321c271101c9de9309f3590be6840a815ba5d495c73ca212.scope\": RecentStats: unable to find data in memory cache]" Nov 02 18:12:05.037226 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.037143 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k69d4\" (UniqueName: \"kubernetes.io/projected/66504ddb-d869-4549-b870-61b42e58e871-kube-api-access-k69d4\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:05.037226 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.037176 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66504ddb-d869-4549-b870-61b42e58e871-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:05.037226 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.037200 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66504ddb-d869-4549-b870-61b42e58e871-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:05.191569 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.191537 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" Nov 02 18:12:05.349313 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.349279 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/719f739a-813b-4f51-9bd4-e523bba953af-kube-api-access-6ffv7\" (UniqueName: \"kubernetes.io/projected/719f739a-813b-4f51-9bd4-e523bba953af-kube-api-access-6ffv7\") pod \"719f739a-813b-4f51-9bd4-e523bba953af\" (UID: \"719f739a-813b-4f51-9bd4-e523bba953af\") " Nov 02 18:12:05.349471 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.349361 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/719f739a-813b-4f51-9bd4-e523bba953af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/719f739a-813b-4f51-9bd4-e523bba953af-metrics-certs\") pod \"719f739a-813b-4f51-9bd4-e523bba953af\" (UID: \"719f739a-813b-4f51-9bd4-e523bba953af\") " Nov 02 18:12:05.349471 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.349402 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/719f739a-813b-4f51-9bd4-e523bba953af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/719f739a-813b-4f51-9bd4-e523bba953af-maas-api-tls\") pod \"719f739a-813b-4f51-9bd4-e523bba953af\" (UID: \"719f739a-813b-4f51-9bd4-e523bba953af\") " Nov 02 18:12:05.349603 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.349577 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/719f739a-813b-4f51-9bd4-e523bba953af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/719f739a-813b-4f51-9bd4-e523bba953af-openshift-service-ca\") pod \"719f739a-813b-4f51-9bd4-e523bba953af\" (UID: \"719f739a-813b-4f51-9bd4-e523bba953af\") " Nov 02 18:12:05.350033 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.349993 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719f739a-813b-4f51-9bd4-e523bba953af-openshift-service-ca" pod "719f739a-813b-4f51-9bd4-e523bba953af" (UID: "719f739a-813b-4f51-9bd4-e523bba953af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:05.350181 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.350113 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/719f739a-813b-4f51-9bd4-e523bba953af-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:05.351637 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.351606 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/719f739a-813b-4f51-9bd4-e523bba953af-metrics-certs" pod "719f739a-813b-4f51-9bd4-e523bba953af" (UID: "719f739a-813b-4f51-9bd4-e523bba953af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:05.351740 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.351641 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/719f739a-813b-4f51-9bd4-e523bba953af-maas-api-tls" pod "719f739a-813b-4f51-9bd4-e523bba953af" (UID: "719f739a-813b-4f51-9bd4-e523bba953af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:05.351995 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.351978 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719f739a-813b-4f51-9bd4-e523bba953af-kube-api-access-6ffv7" pod "719f739a-813b-4f51-9bd4-e523bba953af" (UID: "719f739a-813b-4f51-9bd4-e523bba953af"). InnerVolumeSpecName "kube-api-access-6ffv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:05.435234 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.435190 4860 generic.go:356] "Generic (PLEG): container finished" podID="719f739a-813b-4f51-9bd4-e523bba953af" containerID="6058ad0ba329ef2d321c271101c9de9309f3590be6840a815ba5d495c73ca212" exitCode=0 Nov 02 18:12:05.435448 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.435274 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" Nov 02 18:12:05.435448 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.435309 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" event={"ID":"719f739a-813b-4f51-9bd4-e523bba953af","Type":"ContainerDied","Data":"6058ad0ba329ef2d321c271101c9de9309f3590be6840a815ba5d495c73ca212"} Nov 02 18:12:05.435448 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.435360 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" event={"ID":"719f739a-813b-4f51-9bd4-e523bba953af","Type":"ContainerDied","Data":"5a4fd547dacfffac71541aed9b417023faf2018b8a8ebfbc0d1176c6b8894f8f"} Nov 02 18:12:05.435448 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.435404 4860 scope.go:122] "RemoveContainer" containerID="6058ad0ba329ef2d321c271101c9de9309f3590be6840a815ba5d495c73ca212" Nov 02 18:12:05.437820 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.437792 4860 generic.go:356] "Generic (PLEG): container finished" podID="66504ddb-d869-4549-b870-61b42e58e871" containerID="060ba0b0295fab8763442d2f97fd2a58e32cf4e65d63f7f9b59611601e1cc01c" exitCode=0 Nov 02 18:12:05.437968 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.437895 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" event={"ID":"66504ddb-d869-4549-b870-61b42e58e871","Type":"ContainerDied","Data":"060ba0b0295fab8763442d2f97fd2a58e32cf4e65d63f7f9b59611601e1cc01c"} Nov 02 18:12:05.437968 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.437952 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" event={"ID":"66504ddb-d869-4549-b870-61b42e58e871","Type":"ContainerDied","Data":"154a26262210e0273e8569c2fb88928153bccbc59a752b0533381471bc778e89"} Nov 02 18:12:05.438082 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.437996 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk" Nov 02 18:12:05.452066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.452037 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ffv7\" (UniqueName: \"kubernetes.io/projected/719f739a-813b-4f51-9bd4-e523bba953af-kube-api-access-6ffv7\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:05.452066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.452066 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/719f739a-813b-4f51-9bd4-e523bba953af-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:05.452273 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.452079 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/719f739a-813b-4f51-9bd4-e523bba953af-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:05.453330 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.453300 4860 scope.go:122] "RemoveContainer" containerID="6058ad0ba329ef2d321c271101c9de9309f3590be6840a815ba5d495c73ca212" Nov 02 18:12:05.454196 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:12:05.453959 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6058ad0ba329ef2d321c271101c9de9309f3590be6840a815ba5d495c73ca212\": container with ID starting with 6058ad0ba329ef2d321c271101c9de9309f3590be6840a815ba5d495c73ca212 not found: ID does not exist" containerID="6058ad0ba329ef2d321c271101c9de9309f3590be6840a815ba5d495c73ca212" Nov 02 18:12:05.454196 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.453997 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6058ad0ba329ef2d321c271101c9de9309f3590be6840a815ba5d495c73ca212"} err="failed to get container status \"6058ad0ba329ef2d321c271101c9de9309f3590be6840a815ba5d495c73ca212\": rpc error: code = NotFound desc = could not find container \"6058ad0ba329ef2d321c271101c9de9309f3590be6840a815ba5d495c73ca212\": container with ID starting with 6058ad0ba329ef2d321c271101c9de9309f3590be6840a815ba5d495c73ca212 not found: ID does not exist" Nov 02 18:12:05.454196 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.454023 4860 scope.go:122] "RemoveContainer" containerID="060ba0b0295fab8763442d2f97fd2a58e32cf4e65d63f7f9b59611601e1cc01c" Nov 02 18:12:05.468111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.467988 4860 scope.go:122] "RemoveContainer" containerID="060ba0b0295fab8763442d2f97fd2a58e32cf4e65d63f7f9b59611601e1cc01c" Nov 02 18:12:05.468639 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:12:05.468597 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"060ba0b0295fab8763442d2f97fd2a58e32cf4e65d63f7f9b59611601e1cc01c\": container with ID starting with 060ba0b0295fab8763442d2f97fd2a58e32cf4e65d63f7f9b59611601e1cc01c not found: ID does not exist" containerID="060ba0b0295fab8763442d2f97fd2a58e32cf4e65d63f7f9b59611601e1cc01c" Nov 02 18:12:05.468735 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.468629 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"060ba0b0295fab8763442d2f97fd2a58e32cf4e65d63f7f9b59611601e1cc01c"} err="failed to get container status \"060ba0b0295fab8763442d2f97fd2a58e32cf4e65d63f7f9b59611601e1cc01c\": rpc error: code = NotFound desc = could not find container \"060ba0b0295fab8763442d2f97fd2a58e32cf4e65d63f7f9b59611601e1cc01c\": container with ID starting with 060ba0b0295fab8763442d2f97fd2a58e32cf4e65d63f7f9b59611601e1cc01c not found: ID does not exist" Nov 02 18:12:05.510129 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.510097 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk"] Nov 02 18:12:05.513932 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.513910 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-872354-57d7f6556c-swnrk"] Nov 02 18:12:05.548103 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.548061 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5"] Nov 02 18:12:05.552263 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:05.552236 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5"] Nov 02 18:12:07.291265 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:07.291226 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" podUID="bafb2593-6516-446c-b113-37bff6320729" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:12:07.291811 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:07.291691 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" podUID="75e58f61-43a2-4600-a380-549625283343" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:12:07.503904 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:07.503840 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66504ddb-d869-4549-b870-61b42e58e871" path="/var/lib/kubelet/pods/66504ddb-d869-4549-b870-61b42e58e871/volumes" Nov 02 18:12:07.505202 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:07.505177 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="719f739a-813b-4f51-9bd4-e523bba953af" path="/var/lib/kubelet/pods/719f739a-813b-4f51-9bd4-e523bba953af/volumes" Nov 02 18:12:08.185439 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:08.185391 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5c1da-5bbf8d5bd9-jxmx5" podUID="719f739a-813b-4f51-9bd4-e523bba953af" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.160:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:12:13.542722 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.542671 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" containerID="cri-o://02cba40ea61b2167013c65e8ed83709ca08d0426b48ca2e1a152660cc3bd6b31" gracePeriod=45 Nov 02 18:12:13.571918 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.571886 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" containerID="cri-o://e75c11d78793d51129b7f3f43de7e6bd75b2c3536ba35bebad2d80534aa3d32f" gracePeriod=45 Nov 02 18:12:13.896735 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.896707 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 18:12:13.900707 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.900684 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 18:12:13.992095 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.992058 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-model-cache\") pod \"aee9ada9-06c5-48ab-b6e3-195c1b445270\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " Nov 02 18:12:13.992302 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.992157 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-tmp-dir\") pod \"78574547-89da-456a-a2ca-137fba06a5b3\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " Nov 02 18:12:13.992302 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.992203 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee9ada9-06c5-48ab-b6e3-195c1b445270-kube-api-access-7k8d2\" (UniqueName: \"kubernetes.io/projected/aee9ada9-06c5-48ab-b6e3-195c1b445270-kube-api-access-7k8d2\") pod \"aee9ada9-06c5-48ab-b6e3-195c1b445270\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " Nov 02 18:12:13.992302 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.992238 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78574547-89da-456a-a2ca-137fba06a5b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/78574547-89da-456a-a2ca-137fba06a5b3-tls-certs\") pod \"78574547-89da-456a-a2ca-137fba06a5b3\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " Nov 02 18:12:13.992302 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.992280 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-dshm\") pod \"78574547-89da-456a-a2ca-137fba06a5b3\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " Nov 02 18:12:13.992540 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.992316 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-home\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-home\") pod \"78574547-89da-456a-a2ca-137fba06a5b3\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " Nov 02 18:12:13.992540 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.992359 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-dshm\") pod \"aee9ada9-06c5-48ab-b6e3-195c1b445270\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " Nov 02 18:12:13.992540 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.992448 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-tmp-dir\") pod \"aee9ada9-06c5-48ab-b6e3-195c1b445270\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " Nov 02 18:12:13.992540 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.992524 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-model-cache\") pod \"78574547-89da-456a-a2ca-137fba06a5b3\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " Nov 02 18:12:13.992867 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.992576 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78574547-89da-456a-a2ca-137fba06a5b3-kube-api-access-kkhbm\" (UniqueName: \"kubernetes.io/projected/78574547-89da-456a-a2ca-137fba06a5b3-kube-api-access-kkhbm\") pod \"78574547-89da-456a-a2ca-137fba06a5b3\" (UID: \"78574547-89da-456a-a2ca-137fba06a5b3\") " Nov 02 18:12:13.992867 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.992626 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-home\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-home\") pod \"aee9ada9-06c5-48ab-b6e3-195c1b445270\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " Nov 02 18:12:13.992867 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.992711 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aee9ada9-06c5-48ab-b6e3-195c1b445270-tls-certs\" (UniqueName: \"kubernetes.io/secret/aee9ada9-06c5-48ab-b6e3-195c1b445270-tls-certs\") pod \"aee9ada9-06c5-48ab-b6e3-195c1b445270\" (UID: \"aee9ada9-06c5-48ab-b6e3-195c1b445270\") " Nov 02 18:12:13.993039 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.993015 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-home" pod "78574547-89da-456a-a2ca-137fba06a5b3" (UID: "78574547-89da-456a-a2ca-137fba06a5b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:13.993297 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.993248 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:13.993557 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.993463 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-model-cache" pod "aee9ada9-06c5-48ab-b6e3-195c1b445270" (UID: "aee9ada9-06c5-48ab-b6e3-195c1b445270"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:13.993557 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.993491 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-model-cache" pod "78574547-89da-456a-a2ca-137fba06a5b3" (UID: "78574547-89da-456a-a2ca-137fba06a5b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:13.994349 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.993930 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-tmp-dir" pod "78574547-89da-456a-a2ca-137fba06a5b3" (UID: "78574547-89da-456a-a2ca-137fba06a5b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:13.994349 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.993983 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-tmp-dir" pod "aee9ada9-06c5-48ab-b6e3-195c1b445270" (UID: "aee9ada9-06c5-48ab-b6e3-195c1b445270"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:13.994349 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.994136 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-home" pod "aee9ada9-06c5-48ab-b6e3-195c1b445270" (UID: "aee9ada9-06c5-48ab-b6e3-195c1b445270"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:13.995836 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.995815 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-dshm" pod "78574547-89da-456a-a2ca-137fba06a5b3" (UID: "78574547-89da-456a-a2ca-137fba06a5b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:13.996008 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.995831 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee9ada9-06c5-48ab-b6e3-195c1b445270-tls-certs" pod "aee9ada9-06c5-48ab-b6e3-195c1b445270" (UID: "aee9ada9-06c5-48ab-b6e3-195c1b445270"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:13.996660 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.996633 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78574547-89da-456a-a2ca-137fba06a5b3-kube-api-access-kkhbm" pod "78574547-89da-456a-a2ca-137fba06a5b3" (UID: "78574547-89da-456a-a2ca-137fba06a5b3"). InnerVolumeSpecName "kube-api-access-kkhbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:13.996765 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.996672 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-dshm" pod "aee9ada9-06c5-48ab-b6e3-195c1b445270" (UID: "aee9ada9-06c5-48ab-b6e3-195c1b445270"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:13.996999 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.996974 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78574547-89da-456a-a2ca-137fba06a5b3-tls-certs" pod "78574547-89da-456a-a2ca-137fba06a5b3" (UID: "78574547-89da-456a-a2ca-137fba06a5b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:13.997209 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:13.997186 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee9ada9-06c5-48ab-b6e3-195c1b445270-kube-api-access-7k8d2" pod "aee9ada9-06c5-48ab-b6e3-195c1b445270" (UID: "aee9ada9-06c5-48ab-b6e3-195c1b445270"). InnerVolumeSpecName "kube-api-access-7k8d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:14.094508 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.094476 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.094508 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.094505 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.094508 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.094515 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k8d2\" (UniqueName: \"kubernetes.io/projected/aee9ada9-06c5-48ab-b6e3-195c1b445270-kube-api-access-7k8d2\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.094807 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.094525 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78574547-89da-456a-a2ca-137fba06a5b3-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.094807 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.094535 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.094807 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.094542 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.094807 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.094550 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.094807 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.094557 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78574547-89da-456a-a2ca-137fba06a5b3-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.094807 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.094564 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkhbm\" (UniqueName: \"kubernetes.io/projected/78574547-89da-456a-a2ca-137fba06a5b3-kube-api-access-kkhbm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.094807 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.094572 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee9ada9-06c5-48ab-b6e3-195c1b445270-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.094807 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.094580 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee9ada9-06c5-48ab-b6e3-195c1b445270-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:14.513110 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.513078 4860 generic.go:356] "Generic (PLEG): container finished" podID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerID="02cba40ea61b2167013c65e8ed83709ca08d0426b48ca2e1a152660cc3bd6b31" exitCode=0 Nov 02 18:12:14.513295 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.513132 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" Nov 02 18:12:14.513295 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.513171 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" event={"ID":"aee9ada9-06c5-48ab-b6e3-195c1b445270","Type":"ContainerDied","Data":"02cba40ea61b2167013c65e8ed83709ca08d0426b48ca2e1a152660cc3bd6b31"} Nov 02 18:12:14.513295 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.513215 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" event={"ID":"aee9ada9-06c5-48ab-b6e3-195c1b445270","Type":"ContainerDied","Data":"fd8ff7de8b53a66ef82c9d7a69795b0cfc976ffe25c45b13da09ad49d54a2e7f"} Nov 02 18:12:14.513295 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.513248 4860 scope.go:122] "RemoveContainer" containerID="02cba40ea61b2167013c65e8ed83709ca08d0426b48ca2e1a152660cc3bd6b31" Nov 02 18:12:14.514995 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.514970 4860 generic.go:356] "Generic (PLEG): container finished" podID="78574547-89da-456a-a2ca-137fba06a5b3" containerID="e75c11d78793d51129b7f3f43de7e6bd75b2c3536ba35bebad2d80534aa3d32f" exitCode=0 Nov 02 18:12:14.515101 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.515003 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" event={"ID":"78574547-89da-456a-a2ca-137fba06a5b3","Type":"ContainerDied","Data":"e75c11d78793d51129b7f3f43de7e6bd75b2c3536ba35bebad2d80534aa3d32f"} Nov 02 18:12:14.515101 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.515042 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" event={"ID":"78574547-89da-456a-a2ca-137fba06a5b3","Type":"ContainerDied","Data":"befa37bde5e78a8de8ee488a437bf555477a6e555545a282f4241235749e8dc7"} Nov 02 18:12:14.515101 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.515030 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" Nov 02 18:12:14.526940 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.526915 4860 scope.go:122] "RemoveContainer" containerID="02cba40ea61b2167013c65e8ed83709ca08d0426b48ca2e1a152660cc3bd6b31" Nov 02 18:12:14.527324 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:12:14.527298 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02cba40ea61b2167013c65e8ed83709ca08d0426b48ca2e1a152660cc3bd6b31\": container with ID starting with 02cba40ea61b2167013c65e8ed83709ca08d0426b48ca2e1a152660cc3bd6b31 not found: ID does not exist" containerID="02cba40ea61b2167013c65e8ed83709ca08d0426b48ca2e1a152660cc3bd6b31" Nov 02 18:12:14.527425 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.527335 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02cba40ea61b2167013c65e8ed83709ca08d0426b48ca2e1a152660cc3bd6b31"} err="failed to get container status \"02cba40ea61b2167013c65e8ed83709ca08d0426b48ca2e1a152660cc3bd6b31\": rpc error: code = NotFound desc = could not find container \"02cba40ea61b2167013c65e8ed83709ca08d0426b48ca2e1a152660cc3bd6b31\": container with ID starting with 02cba40ea61b2167013c65e8ed83709ca08d0426b48ca2e1a152660cc3bd6b31 not found: ID does not exist" Nov 02 18:12:14.527425 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.527358 4860 scope.go:122] "RemoveContainer" containerID="e75c11d78793d51129b7f3f43de7e6bd75b2c3536ba35bebad2d80534aa3d32f" Nov 02 18:12:14.537424 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.537406 4860 scope.go:122] "RemoveContainer" containerID="e75c11d78793d51129b7f3f43de7e6bd75b2c3536ba35bebad2d80534aa3d32f" Nov 02 18:12:14.537664 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:12:14.537643 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e75c11d78793d51129b7f3f43de7e6bd75b2c3536ba35bebad2d80534aa3d32f\": container with ID starting with e75c11d78793d51129b7f3f43de7e6bd75b2c3536ba35bebad2d80534aa3d32f not found: ID does not exist" containerID="e75c11d78793d51129b7f3f43de7e6bd75b2c3536ba35bebad2d80534aa3d32f" Nov 02 18:12:14.537718 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.537671 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e75c11d78793d51129b7f3f43de7e6bd75b2c3536ba35bebad2d80534aa3d32f"} err="failed to get container status \"e75c11d78793d51129b7f3f43de7e6bd75b2c3536ba35bebad2d80534aa3d32f\": rpc error: code = NotFound desc = could not find container \"e75c11d78793d51129b7f3f43de7e6bd75b2c3536ba35bebad2d80534aa3d32f\": container with ID starting with e75c11d78793d51129b7f3f43de7e6bd75b2c3536ba35bebad2d80534aa3d32f not found: ID does not exist" Nov 02 18:12:14.567590 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.567561 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6"] Nov 02 18:12:14.579642 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.579617 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6"] Nov 02 18:12:14.621202 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.621169 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg"] Nov 02 18:12:14.624771 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.624738 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg"] Nov 02 18:12:14.777890 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.777792 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/distinct-w2-f0a3b8-kserve-79cdd956c-wq7sg" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:12:14.808142 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:14.808100 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f0a3b8/premium-w2-f0a3b8-kserve-595dcc64f6-4vtt6" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:12:15.500171 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:15.500137 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78574547-89da-456a-a2ca-137fba06a5b3" path="/var/lib/kubelet/pods/78574547-89da-456a-a2ca-137fba06a5b3/volumes" Nov 02 18:12:15.500619 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:15.500605 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" path="/var/lib/kubelet/pods/aee9ada9-06c5-48ab-b6e3-195c1b445270/volumes" Nov 02 18:12:17.290879 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:12:17.290834 4860 logging.go:55] [core] [Channel #6385 SubChannel #6389]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.63:50052", ServerName: "10.132.0.63:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.63:50052: connect: connection refused" Nov 02 18:12:17.292463 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:12:17.292436 4860 logging.go:55] [core] [Channel #6391 SubChannel #6394]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.62:50052", ServerName: "10.132.0.62:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.62:50052: connect: connection refused" Nov 02 18:12:17.503926 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.503900 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" Nov 02 18:12:17.584909 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.573610 4860 generic.go:356] "Generic (PLEG): container finished" podID="bafb2593-6516-446c-b113-37bff6320729" containerID="d0ab35b8d1c48779542d7bf2aed4481f66d9eb405a3b018669c98a3f2902b337" exitCode=0 Nov 02 18:12:17.584909 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.573832 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" event={"ID":"bafb2593-6516-446c-b113-37bff6320729","Type":"ContainerDied","Data":"d0ab35b8d1c48779542d7bf2aed4481f66d9eb405a3b018669c98a3f2902b337"} Nov 02 18:12:17.584909 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.573916 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" event={"ID":"bafb2593-6516-446c-b113-37bff6320729","Type":"ContainerDied","Data":"b38a71891f2d5ad74cc0509065a050a0152fce42064f454449f75787f9c52bae"} Nov 02 18:12:17.584909 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.573952 4860 scope.go:122] "RemoveContainer" containerID="d0ab35b8d1c48779542d7bf2aed4481f66d9eb405a3b018669c98a3f2902b337" Nov 02 18:12:17.584909 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.576118 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" Nov 02 18:12:17.584909 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.580296 4860 generic.go:356] "Generic (PLEG): container finished" podID="75e58f61-43a2-4600-a380-549625283343" containerID="55f7c5a5ab471fc60cd7b430c1d6b6dc35a1ff33131fa7367f753f19d1b10179" exitCode=0 Nov 02 18:12:17.584909 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.580511 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" event={"ID":"75e58f61-43a2-4600-a380-549625283343","Type":"ContainerDied","Data":"55f7c5a5ab471fc60cd7b430c1d6b6dc35a1ff33131fa7367f753f19d1b10179"} Nov 02 18:12:17.584909 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.584561 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" Nov 02 18:12:17.603106 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.599411 4860 scope.go:122] "RemoveContainer" containerID="d0ab35b8d1c48779542d7bf2aed4481f66d9eb405a3b018669c98a3f2902b337" Nov 02 18:12:17.603106 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:12:17.599781 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ab35b8d1c48779542d7bf2aed4481f66d9eb405a3b018669c98a3f2902b337\": container with ID starting with d0ab35b8d1c48779542d7bf2aed4481f66d9eb405a3b018669c98a3f2902b337 not found: ID does not exist" containerID="d0ab35b8d1c48779542d7bf2aed4481f66d9eb405a3b018669c98a3f2902b337" Nov 02 18:12:17.603106 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.599816 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ab35b8d1c48779542d7bf2aed4481f66d9eb405a3b018669c98a3f2902b337"} err="failed to get container status \"d0ab35b8d1c48779542d7bf2aed4481f66d9eb405a3b018669c98a3f2902b337\": rpc error: code = NotFound desc = could not find container \"d0ab35b8d1c48779542d7bf2aed4481f66d9eb405a3b018669c98a3f2902b337\": container with ID starting with d0ab35b8d1c48779542d7bf2aed4481f66d9eb405a3b018669c98a3f2902b337 not found: ID does not exist" Nov 02 18:12:17.637676 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.637639 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e58f61-43a2-4600-a380-549625283343-kube-api-access-lnmk8\" (UniqueName: \"kubernetes.io/projected/75e58f61-43a2-4600-a380-549625283343-kube-api-access-lnmk8\") pod \"75e58f61-43a2-4600-a380-549625283343\" (UID: \"75e58f61-43a2-4600-a380-549625283343\") " Nov 02 18:12:17.637909 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.637704 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e58f61-43a2-4600-a380-549625283343-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e58f61-43a2-4600-a380-549625283343-plugins-config-volume\") pod \"75e58f61-43a2-4600-a380-549625283343\" (UID: \"75e58f61-43a2-4600-a380-549625283343\") " Nov 02 18:12:17.637909 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.637806 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bafb2593-6516-446c-b113-37bff6320729-kube-api-access-5xmv9\" (UniqueName: \"kubernetes.io/projected/bafb2593-6516-446c-b113-37bff6320729-kube-api-access-5xmv9\") pod \"bafb2593-6516-446c-b113-37bff6320729\" (UID: \"bafb2593-6516-446c-b113-37bff6320729\") " Nov 02 18:12:17.637909 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.637867 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bafb2593-6516-446c-b113-37bff6320729-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bafb2593-6516-446c-b113-37bff6320729-plugins-config-volume\") pod \"bafb2593-6516-446c-b113-37bff6320729\" (UID: \"bafb2593-6516-446c-b113-37bff6320729\") " Nov 02 18:12:17.638155 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.638125 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e58f61-43a2-4600-a380-549625283343-plugins-config-volume" pod "75e58f61-43a2-4600-a380-549625283343" (UID: "75e58f61-43a2-4600-a380-549625283343"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:17.638373 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.638341 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bafb2593-6516-446c-b113-37bff6320729-plugins-config-volume" pod "bafb2593-6516-446c-b113-37bff6320729" (UID: "bafb2593-6516-446c-b113-37bff6320729"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:17.638737 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.638715 4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e58f61-43a2-4600-a380-549625283343-plugins-config-volume\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:17.638872 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.638741 4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bafb2593-6516-446c-b113-37bff6320729-plugins-config-volume\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:17.640594 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.640565 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bafb2593-6516-446c-b113-37bff6320729-kube-api-access-5xmv9" pod "bafb2593-6516-446c-b113-37bff6320729" (UID: "bafb2593-6516-446c-b113-37bff6320729"). InnerVolumeSpecName "kube-api-access-5xmv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:17.640706 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.640638 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e58f61-43a2-4600-a380-549625283343-kube-api-access-lnmk8" pod "75e58f61-43a2-4600-a380-549625283343" (UID: "75e58f61-43a2-4600-a380-549625283343"). InnerVolumeSpecName "kube-api-access-lnmk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:17.740238 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.740155 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xmv9\" (UniqueName: \"kubernetes.io/projected/bafb2593-6516-446c-b113-37bff6320729-kube-api-access-5xmv9\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:17.740238 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.740188 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnmk8\" (UniqueName: \"kubernetes.io/projected/75e58f61-43a2-4600-a380-549625283343-kube-api-access-lnmk8\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:17.922786 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.922752 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk"] Nov 02 18:12:17.926832 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:17.926806 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk"] Nov 02 18:12:18.290781 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:18.290743 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-120394-5d6f77dd4c-p4phk" podUID="bafb2593-6516-446c-b113-37bff6320729" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.63:50052\" within 1s: context deadline exceeded" Nov 02 18:12:18.290982 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:18.290876 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" podUID="75e58f61-43a2-4600-a380-549625283343" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.62:50052\" within 1s: context deadline exceeded" Nov 02 18:12:18.290982 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:18.290959 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" Nov 02 18:12:18.589760 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:18.589731 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" Nov 02 18:12:18.589978 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:18.589777 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj" event={"ID":"75e58f61-43a2-4600-a380-549625283343","Type":"ContainerDied","Data":"50c1bb31829e756c31b1c5f1e3d29e8e3fc3d232c71556254908bb6dcc320635"} Nov 02 18:12:18.589978 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:18.589837 4860 scope.go:122] "RemoveContainer" containerID="55f7c5a5ab471fc60cd7b430c1d6b6dc35a1ff33131fa7367f753f19d1b10179" Nov 02 18:12:18.646362 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:18.646326 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj"] Nov 02 18:12:18.656051 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:18.656020 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-120394-68874565f9-wnglj"] Nov 02 18:12:19.500485 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:19.500454 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e58f61-43a2-4600-a380-549625283343" path="/var/lib/kubelet/pods/75e58f61-43a2-4600-a380-549625283343/volumes" Nov 02 18:12:19.500936 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:19.500919 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bafb2593-6516-446c-b113-37bff6320729" path="/var/lib/kubelet/pods/bafb2593-6516-446c-b113-37bff6320729/volumes" Nov 02 18:12:20.685197 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:20.685152 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h"] Nov 02 18:12:21.770386 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:12:21.770348 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:12:25.686124 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:25.686000 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h" podUID="f95eaf1d-668d-405f-bbc1-6be67fbe1fed" containerName="payload-processing" containerID="cri-o://0928ae144f63d074479af700db10032736cc9799383bf76cc273845df6f52d8e" gracePeriod=25 Nov 02 18:12:25.977582 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:25.977541 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h" Nov 02 18:12:26.072981 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:26.072950 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f95eaf1d-668d-405f-bbc1-6be67fbe1fed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95eaf1d-668d-405f-bbc1-6be67fbe1fed-plugins-config-volume\") pod \"f95eaf1d-668d-405f-bbc1-6be67fbe1fed\" (UID: \"f95eaf1d-668d-405f-bbc1-6be67fbe1fed\") " Nov 02 18:12:26.073188 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:26.073010 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95eaf1d-668d-405f-bbc1-6be67fbe1fed-kube-api-access-82r8v\" (UniqueName: \"kubernetes.io/projected/f95eaf1d-668d-405f-bbc1-6be67fbe1fed-kube-api-access-82r8v\") pod \"f95eaf1d-668d-405f-bbc1-6be67fbe1fed\" (UID: \"f95eaf1d-668d-405f-bbc1-6be67fbe1fed\") " Nov 02 18:12:26.073331 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:26.073307 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95eaf1d-668d-405f-bbc1-6be67fbe1fed-plugins-config-volume" pod "f95eaf1d-668d-405f-bbc1-6be67fbe1fed" (UID: "f95eaf1d-668d-405f-bbc1-6be67fbe1fed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:26.075192 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:26.075170 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95eaf1d-668d-405f-bbc1-6be67fbe1fed-kube-api-access-82r8v" pod "f95eaf1d-668d-405f-bbc1-6be67fbe1fed" (UID: "f95eaf1d-668d-405f-bbc1-6be67fbe1fed"). InnerVolumeSpecName "kube-api-access-82r8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:26.174560 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:26.174527 4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95eaf1d-668d-405f-bbc1-6be67fbe1fed-plugins-config-volume\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:26.174560 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:26.174556 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82r8v\" (UniqueName: \"kubernetes.io/projected/f95eaf1d-668d-405f-bbc1-6be67fbe1fed-kube-api-access-82r8v\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:26.675835 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:26.675798 4860 generic.go:356] "Generic (PLEG): container finished" podID="f95eaf1d-668d-405f-bbc1-6be67fbe1fed" containerID="0928ae144f63d074479af700db10032736cc9799383bf76cc273845df6f52d8e" exitCode=0 Nov 02 18:12:26.676048 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:26.675879 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h" Nov 02 18:12:26.676048 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:26.675884 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h" event={"ID":"f95eaf1d-668d-405f-bbc1-6be67fbe1fed","Type":"ContainerDied","Data":"0928ae144f63d074479af700db10032736cc9799383bf76cc273845df6f52d8e"} Nov 02 18:12:26.676048 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:26.675943 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h" event={"ID":"f95eaf1d-668d-405f-bbc1-6be67fbe1fed","Type":"ContainerDied","Data":"085b79d247b534d38e78ae0baf83a1bed4f8908a574c03f8913a042e96ef1d89"} Nov 02 18:12:26.676048 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:26.675967 4860 scope.go:122] "RemoveContainer" containerID="0928ae144f63d074479af700db10032736cc9799383bf76cc273845df6f52d8e" Nov 02 18:12:26.688664 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:26.688456 4860 scope.go:122] "RemoveContainer" containerID="0928ae144f63d074479af700db10032736cc9799383bf76cc273845df6f52d8e" Nov 02 18:12:26.688956 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:12:26.688754 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0928ae144f63d074479af700db10032736cc9799383bf76cc273845df6f52d8e\": container with ID starting with 0928ae144f63d074479af700db10032736cc9799383bf76cc273845df6f52d8e not found: ID does not exist" containerID="0928ae144f63d074479af700db10032736cc9799383bf76cc273845df6f52d8e" Nov 02 18:12:26.688956 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:26.688778 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0928ae144f63d074479af700db10032736cc9799383bf76cc273845df6f52d8e"} err="failed to get container status \"0928ae144f63d074479af700db10032736cc9799383bf76cc273845df6f52d8e\": rpc error: code = NotFound desc = could not find container \"0928ae144f63d074479af700db10032736cc9799383bf76cc273845df6f52d8e\": container with ID starting with 0928ae144f63d074479af700db10032736cc9799383bf76cc273845df6f52d8e not found: ID does not exist" Nov 02 18:12:26.730201 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:26.730168 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h"] Nov 02 18:12:26.733928 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:26.733889 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-557feb-5b4f579dff-v7k5h"] Nov 02 18:12:27.501134 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:27.501099 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f95eaf1d-668d-405f-bbc1-6be67fbe1fed" path="/var/lib/kubelet/pods/f95eaf1d-668d-405f-bbc1-6be67fbe1fed/volumes" Nov 02 18:12:28.465633 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:28.465593 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn"] Nov 02 18:12:28.466184 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:28.465866 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" podUID="9a9aa9e4-1f8b-4512-ae25-51b759bd1c91" containerName="maas-api" containerID="cri-o://8d884b62d3605e5831151ba07bfd94384cdeeead146e4db5f5c56b05777993bd" gracePeriod=30 Nov 02 18:12:28.700276 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:28.700242 4860 generic.go:356] "Generic (PLEG): container finished" podID="9a9aa9e4-1f8b-4512-ae25-51b759bd1c91" containerID="8d884b62d3605e5831151ba07bfd94384cdeeead146e4db5f5c56b05777993bd" exitCode=0 Nov 02 18:12:28.700431 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:28.700315 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" event={"ID":"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91","Type":"ContainerDied","Data":"8d884b62d3605e5831151ba07bfd94384cdeeead146e4db5f5c56b05777993bd"} Nov 02 18:12:28.770375 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:28.770346 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" Nov 02 18:12:28.808512 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:28.808479 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-maas-api-tls\") pod \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\" (UID: \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\") " Nov 02 18:12:28.808685 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:28.808561 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-kube-api-access-sn44b\" (UniqueName: \"kubernetes.io/projected/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-kube-api-access-sn44b\") pod \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\" (UID: \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\") " Nov 02 18:12:28.808685 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:28.808667 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-openshift-service-ca\") pod \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\" (UID: \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\") " Nov 02 18:12:28.808777 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:28.808686 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-metrics-certs\") pod \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\" (UID: \"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91\") " Nov 02 18:12:28.809047 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:28.809017 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-openshift-service-ca" pod "9a9aa9e4-1f8b-4512-ae25-51b759bd1c91" (UID: "9a9aa9e4-1f8b-4512-ae25-51b759bd1c91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:28.810759 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:28.810733 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-kube-api-access-sn44b" pod "9a9aa9e4-1f8b-4512-ae25-51b759bd1c91" (UID: "9a9aa9e4-1f8b-4512-ae25-51b759bd1c91"). InnerVolumeSpecName "kube-api-access-sn44b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:28.810759 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:28.810747 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-maas-api-tls" pod "9a9aa9e4-1f8b-4512-ae25-51b759bd1c91" (UID: "9a9aa9e4-1f8b-4512-ae25-51b759bd1c91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:28.810944 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:28.810828 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-metrics-certs" pod "9a9aa9e4-1f8b-4512-ae25-51b759bd1c91" (UID: "9a9aa9e4-1f8b-4512-ae25-51b759bd1c91"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:28.911247 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:28.911208 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:28.911247 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:28.911247 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:28.911515 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:28.911263 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:28.911515 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:28.911278 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn44b\" (UniqueName: \"kubernetes.io/projected/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91-kube-api-access-sn44b\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:29.385908 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.381016 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8"] Nov 02 18:12:29.386412 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.386357 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" podUID="63743df5-0f1d-4708-8c41-80b2c9344fc5" containerName="maas-api" containerID="cri-o://8754e9920c9fe90cb663ed5ef8b60c9bfc7c08b2d87651fc2eb8896ca754adef" gracePeriod=30 Nov 02 18:12:29.518704 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.518664 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" podUID="63743df5-0f1d-4708-8c41-80b2c9344fc5" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.162:8443/health\": dial tcp 10.132.0.162:8443: connect: connection refused" Nov 02 18:12:29.700237 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.700209 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" Nov 02 18:12:29.709481 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.709451 4860 generic.go:356] "Generic (PLEG): container finished" podID="63743df5-0f1d-4708-8c41-80b2c9344fc5" containerID="8754e9920c9fe90cb663ed5ef8b60c9bfc7c08b2d87651fc2eb8896ca754adef" exitCode=0 Nov 02 18:12:29.709628 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.709493 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" Nov 02 18:12:29.709628 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.709529 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" event={"ID":"63743df5-0f1d-4708-8c41-80b2c9344fc5","Type":"ContainerDied","Data":"8754e9920c9fe90cb663ed5ef8b60c9bfc7c08b2d87651fc2eb8896ca754adef"} Nov 02 18:12:29.709628 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.709578 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8" event={"ID":"63743df5-0f1d-4708-8c41-80b2c9344fc5","Type":"ContainerDied","Data":"cd182ca448e00ace50d5094b35bb7f8c81deff7fcc73b30c91ad36844055d00c"} Nov 02 18:12:29.709628 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.709604 4860 scope.go:122] "RemoveContainer" containerID="8754e9920c9fe90cb663ed5ef8b60c9bfc7c08b2d87651fc2eb8896ca754adef" Nov 02 18:12:29.713289 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.711508 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" Nov 02 18:12:29.713289 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.711533 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn" event={"ID":"9a9aa9e4-1f8b-4512-ae25-51b759bd1c91","Type":"ContainerDied","Data":"38a9f74b44831d367635be12e2f05135edfaf61c5e08c6f5bb12dc09faed940a"} Nov 02 18:12:29.722992 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.722959 4860 scope.go:122] "RemoveContainer" containerID="8754e9920c9fe90cb663ed5ef8b60c9bfc7c08b2d87651fc2eb8896ca754adef" Nov 02 18:12:29.723326 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:12:29.723302 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8754e9920c9fe90cb663ed5ef8b60c9bfc7c08b2d87651fc2eb8896ca754adef\": container with ID starting with 8754e9920c9fe90cb663ed5ef8b60c9bfc7c08b2d87651fc2eb8896ca754adef not found: ID does not exist" containerID="8754e9920c9fe90cb663ed5ef8b60c9bfc7c08b2d87651fc2eb8896ca754adef" Nov 02 18:12:29.723406 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.723335 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8754e9920c9fe90cb663ed5ef8b60c9bfc7c08b2d87651fc2eb8896ca754adef"} err="failed to get container status \"8754e9920c9fe90cb663ed5ef8b60c9bfc7c08b2d87651fc2eb8896ca754adef\": rpc error: code = NotFound desc = could not find container \"8754e9920c9fe90cb663ed5ef8b60c9bfc7c08b2d87651fc2eb8896ca754adef\": container with ID starting with 8754e9920c9fe90cb663ed5ef8b60c9bfc7c08b2d87651fc2eb8896ca754adef not found: ID does not exist" Nov 02 18:12:29.723406 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.723352 4860 scope.go:122] "RemoveContainer" containerID="8d884b62d3605e5831151ba07bfd94384cdeeead146e4db5f5c56b05777993bd" Nov 02 18:12:29.729008 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.728985 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63743df5-0f1d-4708-8c41-80b2c9344fc5-kube-api-access-rd6t9\" (UniqueName: \"kubernetes.io/projected/63743df5-0f1d-4708-8c41-80b2c9344fc5-kube-api-access-rd6t9\") pod \"63743df5-0f1d-4708-8c41-80b2c9344fc5\" (UID: \"63743df5-0f1d-4708-8c41-80b2c9344fc5\") " Nov 02 18:12:29.729132 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.729023 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63743df5-0f1d-4708-8c41-80b2c9344fc5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63743df5-0f1d-4708-8c41-80b2c9344fc5-maas-api-tls\") pod \"63743df5-0f1d-4708-8c41-80b2c9344fc5\" (UID: \"63743df5-0f1d-4708-8c41-80b2c9344fc5\") " Nov 02 18:12:29.729132 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.729060 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63743df5-0f1d-4708-8c41-80b2c9344fc5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63743df5-0f1d-4708-8c41-80b2c9344fc5-openshift-service-ca\") pod \"63743df5-0f1d-4708-8c41-80b2c9344fc5\" (UID: \"63743df5-0f1d-4708-8c41-80b2c9344fc5\") " Nov 02 18:12:29.729263 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.729187 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63743df5-0f1d-4708-8c41-80b2c9344fc5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/63743df5-0f1d-4708-8c41-80b2c9344fc5-metrics-certs\") pod \"63743df5-0f1d-4708-8c41-80b2c9344fc5\" (UID: \"63743df5-0f1d-4708-8c41-80b2c9344fc5\") " Nov 02 18:12:29.729508 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.729481 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63743df5-0f1d-4708-8c41-80b2c9344fc5-openshift-service-ca" pod "63743df5-0f1d-4708-8c41-80b2c9344fc5" (UID: "63743df5-0f1d-4708-8c41-80b2c9344fc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:29.729719 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.729697 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63743df5-0f1d-4708-8c41-80b2c9344fc5-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:29.731512 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.731480 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63743df5-0f1d-4708-8c41-80b2c9344fc5-metrics-certs" pod "63743df5-0f1d-4708-8c41-80b2c9344fc5" (UID: "63743df5-0f1d-4708-8c41-80b2c9344fc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:29.731629 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.731544 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63743df5-0f1d-4708-8c41-80b2c9344fc5-kube-api-access-rd6t9" pod "63743df5-0f1d-4708-8c41-80b2c9344fc5" (UID: "63743df5-0f1d-4708-8c41-80b2c9344fc5"). InnerVolumeSpecName "kube-api-access-rd6t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:29.731949 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.731926 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63743df5-0f1d-4708-8c41-80b2c9344fc5-maas-api-tls" pod "63743df5-0f1d-4708-8c41-80b2c9344fc5" (UID: "63743df5-0f1d-4708-8c41-80b2c9344fc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:29.832919 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.832863 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63743df5-0f1d-4708-8c41-80b2c9344fc5-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:29.832919 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.832907 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd6t9\" (UniqueName: \"kubernetes.io/projected/63743df5-0f1d-4708-8c41-80b2c9344fc5-kube-api-access-rd6t9\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:29.832919 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.832921 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63743df5-0f1d-4708-8c41-80b2c9344fc5-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:29.858622 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.858582 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn"] Nov 02 18:12:29.867654 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:29.867621 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-96768b-5dfb84fc48-hrzvn"] Nov 02 18:12:30.076701 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:30.076666 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8"] Nov 02 18:12:30.080206 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:30.080170 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bb6d2e-9c87bd447-d5gh8"] Nov 02 18:12:30.435332 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:30.432402 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb"] Nov 02 18:12:30.435332 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:30.432647 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" podUID="71b9f322-2bf8-415b-a26a-3723a34c2482" containerName="maas-api" containerID="cri-o://0e4313dc71db38e631122f67a415f8ebf8c51292e3d58787ad7e65a4441a1c77" gracePeriod=30 Nov 02 18:12:30.762475 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:30.762286 4860 generic.go:356] "Generic (PLEG): container finished" podID="71b9f322-2bf8-415b-a26a-3723a34c2482" containerID="0e4313dc71db38e631122f67a415f8ebf8c51292e3d58787ad7e65a4441a1c77" exitCode=0 Nov 02 18:12:30.762985 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:30.762514 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" event={"ID":"71b9f322-2bf8-415b-a26a-3723a34c2482","Type":"ContainerDied","Data":"0e4313dc71db38e631122f67a415f8ebf8c51292e3d58787ad7e65a4441a1c77"} Nov 02 18:12:30.771414 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:30.771345 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" Nov 02 18:12:30.846659 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:30.846616 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71b9f322-2bf8-415b-a26a-3723a34c2482-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b9f322-2bf8-415b-a26a-3723a34c2482-metrics-certs\") pod \"71b9f322-2bf8-415b-a26a-3723a34c2482\" (UID: \"71b9f322-2bf8-415b-a26a-3723a34c2482\") " Nov 02 18:12:30.846858 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:30.846696 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b9f322-2bf8-415b-a26a-3723a34c2482-kube-api-access-rlqlb\" (UniqueName: \"kubernetes.io/projected/71b9f322-2bf8-415b-a26a-3723a34c2482-kube-api-access-rlqlb\") pod \"71b9f322-2bf8-415b-a26a-3723a34c2482\" (UID: \"71b9f322-2bf8-415b-a26a-3723a34c2482\") " Nov 02 18:12:30.846858 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:30.846745 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71b9f322-2bf8-415b-a26a-3723a34c2482-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b9f322-2bf8-415b-a26a-3723a34c2482-maas-api-tls\") pod \"71b9f322-2bf8-415b-a26a-3723a34c2482\" (UID: \"71b9f322-2bf8-415b-a26a-3723a34c2482\") " Nov 02 18:12:30.847559 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:30.847531 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b9f322-2bf8-415b-a26a-3723a34c2482-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b9f322-2bf8-415b-a26a-3723a34c2482-openshift-service-ca\") pod \"71b9f322-2bf8-415b-a26a-3723a34c2482\" (UID: \"71b9f322-2bf8-415b-a26a-3723a34c2482\") " Nov 02 18:12:30.847962 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:30.847933 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b9f322-2bf8-415b-a26a-3723a34c2482-openshift-service-ca" pod "71b9f322-2bf8-415b-a26a-3723a34c2482" (UID: "71b9f322-2bf8-415b-a26a-3723a34c2482"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:30.848529 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:30.848505 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b9f322-2bf8-415b-a26a-3723a34c2482-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:30.849646 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:30.849621 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b9f322-2bf8-415b-a26a-3723a34c2482-maas-api-tls" pod "71b9f322-2bf8-415b-a26a-3723a34c2482" (UID: "71b9f322-2bf8-415b-a26a-3723a34c2482"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:30.849967 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:30.849924 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b9f322-2bf8-415b-a26a-3723a34c2482-kube-api-access-rlqlb" pod "71b9f322-2bf8-415b-a26a-3723a34c2482" (UID: "71b9f322-2bf8-415b-a26a-3723a34c2482"). InnerVolumeSpecName "kube-api-access-rlqlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:30.850824 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:30.850796 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b9f322-2bf8-415b-a26a-3723a34c2482-metrics-certs" pod "71b9f322-2bf8-415b-a26a-3723a34c2482" (UID: "71b9f322-2bf8-415b-a26a-3723a34c2482"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:30.950195 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:30.949531 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b9f322-2bf8-415b-a26a-3723a34c2482-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:30.950195 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:30.949569 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlqlb\" (UniqueName: \"kubernetes.io/projected/71b9f322-2bf8-415b-a26a-3723a34c2482-kube-api-access-rlqlb\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:30.950195 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:30.949584 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b9f322-2bf8-415b-a26a-3723a34c2482-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:31.501442 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:31.501398 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63743df5-0f1d-4708-8c41-80b2c9344fc5" path="/var/lib/kubelet/pods/63743df5-0f1d-4708-8c41-80b2c9344fc5/volumes" Nov 02 18:12:31.501915 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:31.501896 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a9aa9e4-1f8b-4512-ae25-51b759bd1c91" path="/var/lib/kubelet/pods/9a9aa9e4-1f8b-4512-ae25-51b759bd1c91/volumes" Nov 02 18:12:31.775657 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:31.775559 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" Nov 02 18:12:31.775657 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:31.775588 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb" event={"ID":"71b9f322-2bf8-415b-a26a-3723a34c2482","Type":"ContainerDied","Data":"c672c49e06684b4e09687352cbcc36668ccdf0ecc8bec9c1db529643a51af1de"} Nov 02 18:12:31.776244 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:31.775668 4860 scope.go:122] "RemoveContainer" containerID="0e4313dc71db38e631122f67a415f8ebf8c51292e3d58787ad7e65a4441a1c77" Nov 02 18:12:31.858879 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:31.839897 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb"] Nov 02 18:12:31.873593 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:31.873558 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f59572-85b588f9d6-sz4fb"] Nov 02 18:12:33.499286 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:33.499252 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b9f322-2bf8-415b-a26a-3723a34c2482" path="/var/lib/kubelet/pods/71b9f322-2bf8-415b-a26a-3723a34c2482/volumes" Nov 02 18:12:42.096570 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:42.096534 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv"] Nov 02 18:12:42.101548 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:42.096805 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" podUID="69187cd1-8708-46d3-91d9-62b9a9446454" containerName="istio-proxy" containerID="cri-o://96e00330e0c4510c2cbc659ef7d91aa95e1de4096577fe0fa4fbd6592a0f3461" gracePeriod=30 Nov 02 18:12:44.070680 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:44.070640 4860 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:12:44.071194 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:44.070709 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" podUID="69187cd1-8708-46d3-91d9-62b9a9446454" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:12:47.377825 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.377739 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 18:12:47.502657 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.502620 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-credential-socket\") pod \"69187cd1-8708-46d3-91d9-62b9a9446454\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " Nov 02 18:12:47.502657 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.502657 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69187cd1-8708-46d3-91d9-62b9a9446454-kube-api-access-758gj\" (UniqueName: \"kubernetes.io/projected/69187cd1-8708-46d3-91d9-62b9a9446454-kube-api-access-758gj\") pod \"69187cd1-8708-46d3-91d9-62b9a9446454\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " Nov 02 18:12:47.502871 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.502696 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69187cd1-8708-46d3-91d9-62b9a9446454-istio-token\" (UniqueName: \"kubernetes.io/projected/69187cd1-8708-46d3-91d9-62b9a9446454-istio-token\") pod \"69187cd1-8708-46d3-91d9-62b9a9446454\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " Nov 02 18:12:47.502923 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.502835 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69187cd1-8708-46d3-91d9-62b9a9446454-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69187cd1-8708-46d3-91d9-62b9a9446454-istio-podinfo\") pod \"69187cd1-8708-46d3-91d9-62b9a9446454\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " Nov 02 18:12:47.502923 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.502909 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-credential-socket" pod "69187cd1-8708-46d3-91d9-62b9a9446454" (UID: "69187cd1-8708-46d3-91d9-62b9a9446454"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:47.503050 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.502924 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-workload-socket\") pod \"69187cd1-8708-46d3-91d9-62b9a9446454\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " Nov 02 18:12:47.503050 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.502982 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-workload-certs\") pod \"69187cd1-8708-46d3-91d9-62b9a9446454\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " Nov 02 18:12:47.503050 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.503014 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-istio-data\") pod \"69187cd1-8708-46d3-91d9-62b9a9446454\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " Nov 02 18:12:47.503209 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.503080 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-istio-envoy\") pod \"69187cd1-8708-46d3-91d9-62b9a9446454\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " Nov 02 18:12:47.503209 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.503155 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69187cd1-8708-46d3-91d9-62b9a9446454-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69187cd1-8708-46d3-91d9-62b9a9446454-istiod-ca-cert\") pod \"69187cd1-8708-46d3-91d9-62b9a9446454\" (UID: \"69187cd1-8708-46d3-91d9-62b9a9446454\") " Nov 02 18:12:47.503329 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.503192 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-workload-socket" pod "69187cd1-8708-46d3-91d9-62b9a9446454" (UID: "69187cd1-8708-46d3-91d9-62b9a9446454"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:47.503329 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.503256 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-workload-certs" pod "69187cd1-8708-46d3-91d9-62b9a9446454" (UID: "69187cd1-8708-46d3-91d9-62b9a9446454"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:47.503452 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.503404 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-istio-data" pod "69187cd1-8708-46d3-91d9-62b9a9446454" (UID: "69187cd1-8708-46d3-91d9-62b9a9446454"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:47.503671 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.503651 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69187cd1-8708-46d3-91d9-62b9a9446454-istiod-ca-cert" pod "69187cd1-8708-46d3-91d9-62b9a9446454" (UID: "69187cd1-8708-46d3-91d9-62b9a9446454"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:47.503780 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.503718 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:47.503780 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.503741 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:47.503780 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.503754 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:47.503780 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.503768 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:47.505097 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.505072 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69187cd1-8708-46d3-91d9-62b9a9446454-kube-api-access-758gj" pod "69187cd1-8708-46d3-91d9-62b9a9446454" (UID: "69187cd1-8708-46d3-91d9-62b9a9446454"). InnerVolumeSpecName "kube-api-access-758gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:47.505226 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.505207 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-istio-envoy" pod "69187cd1-8708-46d3-91d9-62b9a9446454" (UID: "69187cd1-8708-46d3-91d9-62b9a9446454"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:47.505416 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.505403 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69187cd1-8708-46d3-91d9-62b9a9446454-istio-token" pod "69187cd1-8708-46d3-91d9-62b9a9446454" (UID: "69187cd1-8708-46d3-91d9-62b9a9446454"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:47.505467 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.505446 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69187cd1-8708-46d3-91d9-62b9a9446454-istio-podinfo" pod "69187cd1-8708-46d3-91d9-62b9a9446454" (UID: "69187cd1-8708-46d3-91d9-62b9a9446454"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:12:47.604897 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.604842 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69187cd1-8708-46d3-91d9-62b9a9446454-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:47.604897 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.604897 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69187cd1-8708-46d3-91d9-62b9a9446454-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:47.605126 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.604911 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69187cd1-8708-46d3-91d9-62b9a9446454-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:47.605126 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.604923 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-758gj\" (UniqueName: \"kubernetes.io/projected/69187cd1-8708-46d3-91d9-62b9a9446454-kube-api-access-758gj\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:47.605126 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.604935 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69187cd1-8708-46d3-91d9-62b9a9446454-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:12:47.917683 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.917646 4860 generic.go:356] "Generic (PLEG): container finished" podID="69187cd1-8708-46d3-91d9-62b9a9446454" containerID="96e00330e0c4510c2cbc659ef7d91aa95e1de4096577fe0fa4fbd6592a0f3461" exitCode=0 Nov 02 18:12:47.917937 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.917701 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" Nov 02 18:12:47.917937 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.917750 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" event={"ID":"69187cd1-8708-46d3-91d9-62b9a9446454","Type":"ContainerDied","Data":"96e00330e0c4510c2cbc659ef7d91aa95e1de4096577fe0fa4fbd6592a0f3461"} Nov 02 18:12:47.917937 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.917833 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv" event={"ID":"69187cd1-8708-46d3-91d9-62b9a9446454","Type":"ContainerDied","Data":"686c0617db29ec51d21261a0f1e8bdbf0471e3ac36ae6e543e4a2fe89015c1d4"} Nov 02 18:12:47.917937 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.917904 4860 scope.go:122] "RemoveContainer" containerID="96e00330e0c4510c2cbc659ef7d91aa95e1de4096577fe0fa4fbd6592a0f3461" Nov 02 18:12:47.930956 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.930929 4860 scope.go:122] "RemoveContainer" containerID="96e00330e0c4510c2cbc659ef7d91aa95e1de4096577fe0fa4fbd6592a0f3461" Nov 02 18:12:47.931335 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:12:47.931306 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e00330e0c4510c2cbc659ef7d91aa95e1de4096577fe0fa4fbd6592a0f3461\": container with ID starting with 96e00330e0c4510c2cbc659ef7d91aa95e1de4096577fe0fa4fbd6592a0f3461 not found: ID does not exist" containerID="96e00330e0c4510c2cbc659ef7d91aa95e1de4096577fe0fa4fbd6592a0f3461" Nov 02 18:12:47.931428 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.931348 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e00330e0c4510c2cbc659ef7d91aa95e1de4096577fe0fa4fbd6592a0f3461"} err="failed to get container status \"96e00330e0c4510c2cbc659ef7d91aa95e1de4096577fe0fa4fbd6592a0f3461\": rpc error: code = NotFound desc = could not find container \"96e00330e0c4510c2cbc659ef7d91aa95e1de4096577fe0fa4fbd6592a0f3461\": container with ID starting with 96e00330e0c4510c2cbc659ef7d91aa95e1de4096577fe0fa4fbd6592a0f3461 not found: ID does not exist" Nov 02 18:12:47.974436 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.974404 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv"] Nov 02 18:12:47.977781 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:47.977757 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-td4tv"] Nov 02 18:12:49.500190 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:49.500155 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69187cd1-8708-46d3-91d9-62b9a9446454" path="/var/lib/kubelet/pods/69187cd1-8708-46d3-91d9-62b9a9446454/volumes" Nov 02 18:12:51.770629 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:12:51.770593 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:12:54.878438 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:54.878397 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678"] Nov 02 18:12:59.879687 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:12:59.879647 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678" podUID="e1e2db1f-bd9c-46f0-b913-9533fcf95a4d" containerName="payload-pre-processing" containerID="cri-o://6c9a84e669cf98e98d0b3e37beced94c7b4efa2976da541f540e241b83a982bf" gracePeriod=25 Nov 02 18:13:00.020073 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:00.020032 4860 generic.go:356] "Generic (PLEG): container finished" podID="e1e2db1f-bd9c-46f0-b913-9533fcf95a4d" containerID="6c9a84e669cf98e98d0b3e37beced94c7b4efa2976da541f540e241b83a982bf" exitCode=0 Nov 02 18:13:00.020230 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:00.020101 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678" event={"ID":"e1e2db1f-bd9c-46f0-b913-9533fcf95a4d","Type":"ContainerDied","Data":"6c9a84e669cf98e98d0b3e37beced94c7b4efa2976da541f540e241b83a982bf"} Nov 02 18:13:00.164074 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:00.164047 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678" Nov 02 18:13:00.292594 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:00.292563 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1e2db1f-bd9c-46f0-b913-9533fcf95a4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e2db1f-bd9c-46f0-b913-9533fcf95a4d-plugins-config-volume\") pod \"e1e2db1f-bd9c-46f0-b913-9533fcf95a4d\" (UID: \"e1e2db1f-bd9c-46f0-b913-9533fcf95a4d\") " Nov 02 18:13:00.292770 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:00.292637 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e2db1f-bd9c-46f0-b913-9533fcf95a4d-kube-api-access-s7z4s\" (UniqueName: \"kubernetes.io/projected/e1e2db1f-bd9c-46f0-b913-9533fcf95a4d-kube-api-access-s7z4s\") pod \"e1e2db1f-bd9c-46f0-b913-9533fcf95a4d\" (UID: \"e1e2db1f-bd9c-46f0-b913-9533fcf95a4d\") " Nov 02 18:13:00.293045 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:00.293015 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e2db1f-bd9c-46f0-b913-9533fcf95a4d-plugins-config-volume" pod "e1e2db1f-bd9c-46f0-b913-9533fcf95a4d" (UID: "e1e2db1f-bd9c-46f0-b913-9533fcf95a4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:00.293329 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:00.293312 4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e2db1f-bd9c-46f0-b913-9533fcf95a4d-plugins-config-volume\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:00.294935 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:00.294906 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e2db1f-bd9c-46f0-b913-9533fcf95a4d-kube-api-access-s7z4s" pod "e1e2db1f-bd9c-46f0-b913-9533fcf95a4d" (UID: "e1e2db1f-bd9c-46f0-b913-9533fcf95a4d"). InnerVolumeSpecName "kube-api-access-s7z4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:00.394615 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:00.394537 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7z4s\" (UniqueName: \"kubernetes.io/projected/e1e2db1f-bd9c-46f0-b913-9533fcf95a4d-kube-api-access-s7z4s\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:01.028507 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:01.028478 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678" Nov 02 18:13:01.028507 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:01.028490 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678" event={"ID":"e1e2db1f-bd9c-46f0-b913-9533fcf95a4d","Type":"ContainerDied","Data":"db341a5bea5df22935f0045a692cac0ea482a73f182f65fb10d702282ad2f9ff"} Nov 02 18:13:01.029046 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:01.028555 4860 scope.go:122] "RemoveContainer" containerID="6c9a84e669cf98e98d0b3e37beced94c7b4efa2976da541f540e241b83a982bf" Nov 02 18:13:01.084345 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:01.084313 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678"] Nov 02 18:13:01.096465 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:01.096440 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-227939-645c4b85f5-wn678"] Nov 02 18:13:01.499370 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:01.499339 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1e2db1f-bd9c-46f0-b913-9533fcf95a4d" path="/var/lib/kubelet/pods/e1e2db1f-bd9c-46f0-b913-9533fcf95a4d/volumes" Nov 02 18:13:04.354234 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:04.354199 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj"] Nov 02 18:13:04.354687 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:04.354442 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" podUID="42544ef4-8181-4c6e-bbd8-337a63fc0412" containerName="maas-api" containerID="cri-o://4acfd9abf14babf6e4dfb3c1347b27b38ca74931e7d643864f6d8bb9526ccb60" gracePeriod=30 Nov 02 18:13:04.684049 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:04.684018 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" Nov 02 18:13:04.766243 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:04.766204 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42544ef4-8181-4c6e-bbd8-337a63fc0412-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42544ef4-8181-4c6e-bbd8-337a63fc0412-openshift-service-ca\") pod \"42544ef4-8181-4c6e-bbd8-337a63fc0412\" (UID: \"42544ef4-8181-4c6e-bbd8-337a63fc0412\") " Nov 02 18:13:04.766455 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:04.766271 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42544ef4-8181-4c6e-bbd8-337a63fc0412-metrics-certs\" (UniqueName: \"kubernetes.io/secret/42544ef4-8181-4c6e-bbd8-337a63fc0412-metrics-certs\") pod \"42544ef4-8181-4c6e-bbd8-337a63fc0412\" (UID: \"42544ef4-8181-4c6e-bbd8-337a63fc0412\") " Nov 02 18:13:04.766455 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:04.766421 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42544ef4-8181-4c6e-bbd8-337a63fc0412-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42544ef4-8181-4c6e-bbd8-337a63fc0412-maas-api-tls\") pod \"42544ef4-8181-4c6e-bbd8-337a63fc0412\" (UID: \"42544ef4-8181-4c6e-bbd8-337a63fc0412\") " Nov 02 18:13:04.766586 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:04.766453 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42544ef4-8181-4c6e-bbd8-337a63fc0412-kube-api-access-vljrt\" (UniqueName: \"kubernetes.io/projected/42544ef4-8181-4c6e-bbd8-337a63fc0412-kube-api-access-vljrt\") pod \"42544ef4-8181-4c6e-bbd8-337a63fc0412\" (UID: \"42544ef4-8181-4c6e-bbd8-337a63fc0412\") " Nov 02 18:13:04.767947 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:04.767901 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42544ef4-8181-4c6e-bbd8-337a63fc0412-openshift-service-ca" pod "42544ef4-8181-4c6e-bbd8-337a63fc0412" (UID: "42544ef4-8181-4c6e-bbd8-337a63fc0412"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:04.771320 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:04.771286 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42544ef4-8181-4c6e-bbd8-337a63fc0412-kube-api-access-vljrt" pod "42544ef4-8181-4c6e-bbd8-337a63fc0412" (UID: "42544ef4-8181-4c6e-bbd8-337a63fc0412"). InnerVolumeSpecName "kube-api-access-vljrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:04.771656 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:04.771634 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42544ef4-8181-4c6e-bbd8-337a63fc0412-maas-api-tls" pod "42544ef4-8181-4c6e-bbd8-337a63fc0412" (UID: "42544ef4-8181-4c6e-bbd8-337a63fc0412"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:04.772140 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:04.772108 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42544ef4-8181-4c6e-bbd8-337a63fc0412-metrics-certs" pod "42544ef4-8181-4c6e-bbd8-337a63fc0412" (UID: "42544ef4-8181-4c6e-bbd8-337a63fc0412"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:04.868290 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:04.868253 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vljrt\" (UniqueName: \"kubernetes.io/projected/42544ef4-8181-4c6e-bbd8-337a63fc0412-kube-api-access-vljrt\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:04.868290 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:04.868290 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42544ef4-8181-4c6e-bbd8-337a63fc0412-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:04.868574 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:04.868307 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42544ef4-8181-4c6e-bbd8-337a63fc0412-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:04.868574 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:04.868321 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42544ef4-8181-4c6e-bbd8-337a63fc0412-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:05.064084 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:05.064043 4860 generic.go:356] "Generic (PLEG): container finished" podID="42544ef4-8181-4c6e-bbd8-337a63fc0412" containerID="4acfd9abf14babf6e4dfb3c1347b27b38ca74931e7d643864f6d8bb9526ccb60" exitCode=0 Nov 02 18:13:05.064301 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:05.064099 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" Nov 02 18:13:05.064301 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:05.064131 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" event={"ID":"42544ef4-8181-4c6e-bbd8-337a63fc0412","Type":"ContainerDied","Data":"4acfd9abf14babf6e4dfb3c1347b27b38ca74931e7d643864f6d8bb9526ccb60"} Nov 02 18:13:05.064301 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:05.064202 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj" event={"ID":"42544ef4-8181-4c6e-bbd8-337a63fc0412","Type":"ContainerDied","Data":"c8a42c4c649bf7741672ecbff7c5bd3627397d9502ce8458b204960531fdc335"} Nov 02 18:13:05.064301 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:05.064232 4860 scope.go:122] "RemoveContainer" containerID="4acfd9abf14babf6e4dfb3c1347b27b38ca74931e7d643864f6d8bb9526ccb60" Nov 02 18:13:05.076341 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:05.076315 4860 scope.go:122] "RemoveContainer" containerID="4acfd9abf14babf6e4dfb3c1347b27b38ca74931e7d643864f6d8bb9526ccb60" Nov 02 18:13:05.076657 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:13:05.076631 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4acfd9abf14babf6e4dfb3c1347b27b38ca74931e7d643864f6d8bb9526ccb60\": container with ID starting with 4acfd9abf14babf6e4dfb3c1347b27b38ca74931e7d643864f6d8bb9526ccb60 not found: ID does not exist" containerID="4acfd9abf14babf6e4dfb3c1347b27b38ca74931e7d643864f6d8bb9526ccb60" Nov 02 18:13:05.076732 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:05.076669 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4acfd9abf14babf6e4dfb3c1347b27b38ca74931e7d643864f6d8bb9526ccb60"} err="failed to get container status \"4acfd9abf14babf6e4dfb3c1347b27b38ca74931e7d643864f6d8bb9526ccb60\": rpc error: code = NotFound desc = could not find container \"4acfd9abf14babf6e4dfb3c1347b27b38ca74931e7d643864f6d8bb9526ccb60\": container with ID starting with 4acfd9abf14babf6e4dfb3c1347b27b38ca74931e7d643864f6d8bb9526ccb60 not found: ID does not exist" Nov 02 18:13:05.118437 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:05.118404 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj"] Nov 02 18:13:05.130121 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:05.130093 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-c810ca-5b5d897ddb-25ctj"] Nov 02 18:13:05.500385 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:05.500307 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42544ef4-8181-4c6e-bbd8-337a63fc0412" path="/var/lib/kubelet/pods/42544ef4-8181-4c6e-bbd8-337a63fc0412/volumes" Nov 02 18:13:12.610556 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:12.610513 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg"] Nov 02 18:13:12.645698 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:12.645669 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d"] Nov 02 18:13:14.505271 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:14.505237 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6"] Nov 02 18:13:17.196612 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:17.196568 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52"] Nov 02 18:13:17.208635 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:17.208591 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg"] Nov 02 18:13:17.212089 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:17.212063 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf"] Nov 02 18:13:17.223903 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:17.223875 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq"] Nov 02 18:13:18.318321 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:18.318291 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm"] Nov 02 18:13:18.353154 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:18.353118 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl"] Nov 02 18:13:19.327731 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:19.327696 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79"] Nov 02 18:13:20.343348 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:20.343308 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd"] Nov 02 18:13:20.379532 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:20.379496 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76"] Nov 02 18:13:21.771518 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:13:21.771483 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:13:23.318793 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:23.318749 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm" podUID="980e42a5-a602-46ea-842d-6b6914e1b870" containerName="payload-processing" containerID="cri-o://23651af746e5a53fcd157d294c2213c9804a085898f5cde03154f455fe51000b" gracePeriod=25 Nov 02 18:13:23.354038 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:23.353972 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl" podUID="63359d54-2671-4cbc-9033-cc1716d65580" containerName="payload-pre-processing" containerID="cri-o://4b40d12ad66cfa3569eb26aba53a7609344938b410d508b0a2e0afcdc5c8e16f" gracePeriod=25 Nov 02 18:13:23.665421 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:23.665019 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm" Nov 02 18:13:23.685478 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:23.685446 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl" Nov 02 18:13:23.705699 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:23.705668 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/980e42a5-a602-46ea-842d-6b6914e1b870-kube-api-access-zj5vc\" (UniqueName: \"kubernetes.io/projected/980e42a5-a602-46ea-842d-6b6914e1b870-kube-api-access-zj5vc\") pod \"980e42a5-a602-46ea-842d-6b6914e1b870\" (UID: \"980e42a5-a602-46ea-842d-6b6914e1b870\") " Nov 02 18:13:23.705882 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:23.705834 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/980e42a5-a602-46ea-842d-6b6914e1b870-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/980e42a5-a602-46ea-842d-6b6914e1b870-plugins-config-volume\") pod \"980e42a5-a602-46ea-842d-6b6914e1b870\" (UID: \"980e42a5-a602-46ea-842d-6b6914e1b870\") " Nov 02 18:13:23.706202 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:23.706175 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980e42a5-a602-46ea-842d-6b6914e1b870-plugins-config-volume" pod "980e42a5-a602-46ea-842d-6b6914e1b870" (UID: "980e42a5-a602-46ea-842d-6b6914e1b870"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:23.706445 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:23.706417 4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/980e42a5-a602-46ea-842d-6b6914e1b870-plugins-config-volume\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:23.707915 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:23.707892 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980e42a5-a602-46ea-842d-6b6914e1b870-kube-api-access-zj5vc" pod "980e42a5-a602-46ea-842d-6b6914e1b870" (UID: "980e42a5-a602-46ea-842d-6b6914e1b870"). InnerVolumeSpecName "kube-api-access-zj5vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:23.807060 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:23.807025 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63359d54-2671-4cbc-9033-cc1716d65580-kube-api-access-hzlbq\" (UniqueName: \"kubernetes.io/projected/63359d54-2671-4cbc-9033-cc1716d65580-kube-api-access-hzlbq\") pod \"63359d54-2671-4cbc-9033-cc1716d65580\" (UID: \"63359d54-2671-4cbc-9033-cc1716d65580\") " Nov 02 18:13:23.807273 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:23.807135 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63359d54-2671-4cbc-9033-cc1716d65580-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63359d54-2671-4cbc-9033-cc1716d65580-plugins-config-volume\") pod \"63359d54-2671-4cbc-9033-cc1716d65580\" (UID: \"63359d54-2671-4cbc-9033-cc1716d65580\") " Nov 02 18:13:23.807462 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:23.807445 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj5vc\" (UniqueName: \"kubernetes.io/projected/980e42a5-a602-46ea-842d-6b6914e1b870-kube-api-access-zj5vc\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:23.807558 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:23.807519 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63359d54-2671-4cbc-9033-cc1716d65580-plugins-config-volume" pod "63359d54-2671-4cbc-9033-cc1716d65580" (UID: "63359d54-2671-4cbc-9033-cc1716d65580"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:23.809374 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:23.809349 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63359d54-2671-4cbc-9033-cc1716d65580-kube-api-access-hzlbq" pod "63359d54-2671-4cbc-9033-cc1716d65580" (UID: "63359d54-2671-4cbc-9033-cc1716d65580"). InnerVolumeSpecName "kube-api-access-hzlbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:23.909062 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:23.908977 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzlbq\" (UniqueName: \"kubernetes.io/projected/63359d54-2671-4cbc-9033-cc1716d65580-kube-api-access-hzlbq\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:23.909062 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:23.909009 4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63359d54-2671-4cbc-9033-cc1716d65580-plugins-config-volume\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:24.229063 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:24.228981 4860 generic.go:356] "Generic (PLEG): container finished" podID="63359d54-2671-4cbc-9033-cc1716d65580" containerID="4b40d12ad66cfa3569eb26aba53a7609344938b410d508b0a2e0afcdc5c8e16f" exitCode=0 Nov 02 18:13:24.229286 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:24.229092 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl" event={"ID":"63359d54-2671-4cbc-9033-cc1716d65580","Type":"ContainerDied","Data":"4b40d12ad66cfa3569eb26aba53a7609344938b410d508b0a2e0afcdc5c8e16f"} Nov 02 18:13:24.229286 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:24.229161 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl" event={"ID":"63359d54-2671-4cbc-9033-cc1716d65580","Type":"ContainerDied","Data":"d7fdc8a53ff9c5acb20d3de679575b9f4334fbe267be6b58652ee6fecaf0cf24"} Nov 02 18:13:24.229286 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:24.229182 4860 scope.go:122] "RemoveContainer" containerID="4b40d12ad66cfa3569eb26aba53a7609344938b410d508b0a2e0afcdc5c8e16f" Nov 02 18:13:24.229481 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:24.229334 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl" Nov 02 18:13:24.233393 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:24.233360 4860 generic.go:356] "Generic (PLEG): container finished" podID="980e42a5-a602-46ea-842d-6b6914e1b870" containerID="23651af746e5a53fcd157d294c2213c9804a085898f5cde03154f455fe51000b" exitCode=0 Nov 02 18:13:24.233619 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:24.233601 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm" event={"ID":"980e42a5-a602-46ea-842d-6b6914e1b870","Type":"ContainerDied","Data":"23651af746e5a53fcd157d294c2213c9804a085898f5cde03154f455fe51000b"} Nov 02 18:13:24.233696 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:24.233648 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm" event={"ID":"980e42a5-a602-46ea-842d-6b6914e1b870","Type":"ContainerDied","Data":"a7b8f0a5793605ea4afdc92ff2696051117f90e0996b4e42c7a7e77e21a960de"} Nov 02 18:13:24.233802 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:24.233787 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm" Nov 02 18:13:24.244022 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:24.243995 4860 scope.go:122] "RemoveContainer" containerID="4b40d12ad66cfa3569eb26aba53a7609344938b410d508b0a2e0afcdc5c8e16f" Nov 02 18:13:24.244550 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:13:24.244521 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b40d12ad66cfa3569eb26aba53a7609344938b410d508b0a2e0afcdc5c8e16f\": container with ID starting with 4b40d12ad66cfa3569eb26aba53a7609344938b410d508b0a2e0afcdc5c8e16f not found: ID does not exist" containerID="4b40d12ad66cfa3569eb26aba53a7609344938b410d508b0a2e0afcdc5c8e16f" Nov 02 18:13:24.244659 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:24.244561 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b40d12ad66cfa3569eb26aba53a7609344938b410d508b0a2e0afcdc5c8e16f"} err="failed to get container status \"4b40d12ad66cfa3569eb26aba53a7609344938b410d508b0a2e0afcdc5c8e16f\": rpc error: code = NotFound desc = could not find container \"4b40d12ad66cfa3569eb26aba53a7609344938b410d508b0a2e0afcdc5c8e16f\": container with ID starting with 4b40d12ad66cfa3569eb26aba53a7609344938b410d508b0a2e0afcdc5c8e16f not found: ID does not exist" Nov 02 18:13:24.244659 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:24.244588 4860 scope.go:122] "RemoveContainer" containerID="23651af746e5a53fcd157d294c2213c9804a085898f5cde03154f455fe51000b" Nov 02 18:13:24.255194 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:24.255169 4860 scope.go:122] "RemoveContainer" containerID="23651af746e5a53fcd157d294c2213c9804a085898f5cde03154f455fe51000b" Nov 02 18:13:24.255530 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:13:24.255508 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23651af746e5a53fcd157d294c2213c9804a085898f5cde03154f455fe51000b\": container with ID starting with 23651af746e5a53fcd157d294c2213c9804a085898f5cde03154f455fe51000b not found: ID does not exist" containerID="23651af746e5a53fcd157d294c2213c9804a085898f5cde03154f455fe51000b" Nov 02 18:13:24.255595 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:24.255541 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23651af746e5a53fcd157d294c2213c9804a085898f5cde03154f455fe51000b"} err="failed to get container status \"23651af746e5a53fcd157d294c2213c9804a085898f5cde03154f455fe51000b\": rpc error: code = NotFound desc = could not find container \"23651af746e5a53fcd157d294c2213c9804a085898f5cde03154f455fe51000b\": container with ID starting with 23651af746e5a53fcd157d294c2213c9804a085898f5cde03154f455fe51000b not found: ID does not exist" Nov 02 18:13:24.300814 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:24.300778 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl"] Nov 02 18:13:24.306783 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:24.306758 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7df526-7c8dcb999f-v88gl"] Nov 02 18:13:24.341495 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:24.341462 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm"] Nov 02 18:13:24.345133 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:24.345104 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7df526-69c44fbf87-kqltm"] Nov 02 18:13:25.029048 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:25.029008 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr"] Nov 02 18:13:25.060144 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:25.060106 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t"] Nov 02 18:13:25.343933 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:25.343889 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" podUID="a27864d8-7810-45a3-8cf1-ccc0033b4268" containerName="payload-processing" containerID="cri-o://56e35e030b02e0bbeb93edf23cbd27ee008616642485520516f30425cff477b1" gracePeriod=25 Nov 02 18:13:25.380467 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:25.380419 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76" podUID="caee9964-de70-4471-9013-90b120d20fa1" containerName="payload-pre-processing" containerID="cri-o://2d3be95a5d3ce1bf9e41a790719e393a77fdfdcfcb3e311a566d7d4d0d4d670f" gracePeriod=25 Nov 02 18:13:25.499148 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:25.499124 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63359d54-2671-4cbc-9033-cc1716d65580" path="/var/lib/kubelet/pods/63359d54-2671-4cbc-9033-cc1716d65580/volumes" Nov 02 18:13:25.499546 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:25.499532 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="980e42a5-a602-46ea-842d-6b6914e1b870" path="/var/lib/kubelet/pods/980e42a5-a602-46ea-842d-6b6914e1b870/volumes" Nov 02 18:13:25.686630 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:25.686603 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76" Nov 02 18:13:25.756498 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:25.756452 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caee9964-de70-4471-9013-90b120d20fa1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caee9964-de70-4471-9013-90b120d20fa1-plugins-config-volume\") pod \"caee9964-de70-4471-9013-90b120d20fa1\" (UID: \"caee9964-de70-4471-9013-90b120d20fa1\") " Nov 02 18:13:25.756693 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:25.756508 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caee9964-de70-4471-9013-90b120d20fa1-kube-api-access-fsbk5\" (UniqueName: \"kubernetes.io/projected/caee9964-de70-4471-9013-90b120d20fa1-kube-api-access-fsbk5\") pod \"caee9964-de70-4471-9013-90b120d20fa1\" (UID: \"caee9964-de70-4471-9013-90b120d20fa1\") " Nov 02 18:13:25.756893 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:25.756837 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caee9964-de70-4471-9013-90b120d20fa1-plugins-config-volume" pod "caee9964-de70-4471-9013-90b120d20fa1" (UID: "caee9964-de70-4471-9013-90b120d20fa1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:25.757054 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:25.757036 4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caee9964-de70-4471-9013-90b120d20fa1-plugins-config-volume\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:25.758964 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:25.758838 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caee9964-de70-4471-9013-90b120d20fa1-kube-api-access-fsbk5" pod "caee9964-de70-4471-9013-90b120d20fa1" (UID: "caee9964-de70-4471-9013-90b120d20fa1"). InnerVolumeSpecName "kube-api-access-fsbk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:25.858312 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:25.858271 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsbk5\" (UniqueName: \"kubernetes.io/projected/caee9964-de70-4471-9013-90b120d20fa1-kube-api-access-fsbk5\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:26.252202 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:26.252164 4860 generic.go:356] "Generic (PLEG): container finished" podID="caee9964-de70-4471-9013-90b120d20fa1" containerID="2d3be95a5d3ce1bf9e41a790719e393a77fdfdcfcb3e311a566d7d4d0d4d670f" exitCode=0 Nov 02 18:13:26.252391 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:26.252208 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76" Nov 02 18:13:26.252391 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:26.252247 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76" event={"ID":"caee9964-de70-4471-9013-90b120d20fa1","Type":"ContainerDied","Data":"2d3be95a5d3ce1bf9e41a790719e393a77fdfdcfcb3e311a566d7d4d0d4d670f"} Nov 02 18:13:26.252391 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:26.252296 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76" event={"ID":"caee9964-de70-4471-9013-90b120d20fa1","Type":"ContainerDied","Data":"2fd6ef94e619bb911680715250479cb1cf51269bd04814bddd57d8ffe4044d99"} Nov 02 18:13:26.252391 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:26.252320 4860 scope.go:122] "RemoveContainer" containerID="2d3be95a5d3ce1bf9e41a790719e393a77fdfdcfcb3e311a566d7d4d0d4d670f" Nov 02 18:13:26.268279 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:26.267603 4860 scope.go:122] "RemoveContainer" containerID="2d3be95a5d3ce1bf9e41a790719e393a77fdfdcfcb3e311a566d7d4d0d4d670f" Nov 02 18:13:26.268279 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:13:26.267997 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d3be95a5d3ce1bf9e41a790719e393a77fdfdcfcb3e311a566d7d4d0d4d670f\": container with ID starting with 2d3be95a5d3ce1bf9e41a790719e393a77fdfdcfcb3e311a566d7d4d0d4d670f not found: ID does not exist" containerID="2d3be95a5d3ce1bf9e41a790719e393a77fdfdcfcb3e311a566d7d4d0d4d670f" Nov 02 18:13:26.268279 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:26.268034 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d3be95a5d3ce1bf9e41a790719e393a77fdfdcfcb3e311a566d7d4d0d4d670f"} err="failed to get container status \"2d3be95a5d3ce1bf9e41a790719e393a77fdfdcfcb3e311a566d7d4d0d4d670f\": rpc error: code = NotFound desc = could not find container \"2d3be95a5d3ce1bf9e41a790719e393a77fdfdcfcb3e311a566d7d4d0d4d670f\": container with ID starting with 2d3be95a5d3ce1bf9e41a790719e393a77fdfdcfcb3e311a566d7d4d0d4d670f not found: ID does not exist" Nov 02 18:13:26.309015 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:26.308977 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76"] Nov 02 18:13:26.320506 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:26.320464 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5eedab-857b74f4cf-6zz76"] Nov 02 18:13:27.292015 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:27.291971 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" podUID="a27864d8-7810-45a3-8cf1-ccc0033b4268" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:13:27.499918 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:27.499882 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caee9964-de70-4471-9013-90b120d20fa1" path="/var/lib/kubelet/pods/caee9964-de70-4471-9013-90b120d20fa1/volumes" Nov 02 18:13:27.633470 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:27.633431 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" containerID="cri-o://a65c79f3e530878efb797ed99d60140f0c380257673787b01bb59cc13afb0228" gracePeriod=45 Nov 02 18:13:27.667538 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:27.667494 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" containerID="cri-o://e5d7b58377638c9c2714dfa776c4a2700db884b20e1de81751eb49b3f52d9bfd" gracePeriod=45 Nov 02 18:13:27.683111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:27.683084 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": dial tcp 10.132.0.98:8000: connect: connection refused" Nov 02 18:13:27.939922 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:27.939893 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 18:13:27.972309 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:27.972278 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 18:13:28.089163 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.089123 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-tls-certs\") pod \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " Nov 02 18:13:28.089340 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.089251 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-home\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-home\") pod \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " Nov 02 18:13:28.089340 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.089288 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-home\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-home\") pod \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " Nov 02 18:13:28.089340 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.089329 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-dshm\") pod \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " Nov 02 18:13:28.089523 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.089358 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-tls-certs\" (UniqueName: \"kubernetes.io/secret/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-tls-certs\") pod \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " Nov 02 18:13:28.089523 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.089392 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-tmp-dir\") pod \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " Nov 02 18:13:28.089523 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.089455 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-kube-api-access-b5xgb\" (UniqueName: \"kubernetes.io/projected/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-kube-api-access-b5xgb\") pod \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " Nov 02 18:13:28.089523 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.089485 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-tmp-dir\") pod \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " Nov 02 18:13:28.089810 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.089558 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-home" pod "12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" (UID: "12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:28.089810 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.089584 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-dshm\") pod \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " Nov 02 18:13:28.089810 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.089685 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-kube-api-access-vqws9\" (UniqueName: \"kubernetes.io/projected/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-kube-api-access-vqws9\") pod \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " Nov 02 18:13:28.089810 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.089751 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-model-cache\") pod \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\" (UID: \"ebd4fcc7-38eb-40ff-b389-8b1fce29d132\") " Nov 02 18:13:28.089810 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.089780 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-home" pod "ebd4fcc7-38eb-40ff-b389-8b1fce29d132" (UID: "ebd4fcc7-38eb-40ff-b389-8b1fce29d132"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:28.089810 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.089795 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-model-cache\") pod \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\" (UID: \"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5\") " Nov 02 18:13:28.090829 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.090017 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-tmp-dir" pod "12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" (UID: "12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:28.091231 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.091202 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-tmp-dir" pod "ebd4fcc7-38eb-40ff-b389-8b1fce29d132" (UID: "ebd4fcc7-38eb-40ff-b389-8b1fce29d132"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:28.091427 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.091402 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-model-cache" pod "ebd4fcc7-38eb-40ff-b389-8b1fce29d132" (UID: "ebd4fcc7-38eb-40ff-b389-8b1fce29d132"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:28.091800 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.091776 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.092115 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.092094 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.092202 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.092118 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.092202 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.092131 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.092202 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.092143 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.092202 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.091915 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-model-cache" pod "12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" (UID: "12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:28.092202 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.092003 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-tls-certs" pod "ebd4fcc7-38eb-40ff-b389-8b1fce29d132" (UID: "ebd4fcc7-38eb-40ff-b389-8b1fce29d132"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:28.092202 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.092001 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-dshm" pod "12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" (UID: "12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:28.092552 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.092423 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-tls-certs" pod "12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" (UID: "12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:28.093756 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.093730 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-dshm" pod "ebd4fcc7-38eb-40ff-b389-8b1fce29d132" (UID: "ebd4fcc7-38eb-40ff-b389-8b1fce29d132"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:28.094313 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.094291 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-kube-api-access-b5xgb" pod "ebd4fcc7-38eb-40ff-b389-8b1fce29d132" (UID: "ebd4fcc7-38eb-40ff-b389-8b1fce29d132"). InnerVolumeSpecName "kube-api-access-b5xgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:28.095131 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.095113 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-kube-api-access-vqws9" pod "12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" (UID: "12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5"). InnerVolumeSpecName "kube-api-access-vqws9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:28.193634 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.193536 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.193634 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.193572 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.193634 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.193590 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5xgb\" (UniqueName: \"kubernetes.io/projected/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-kube-api-access-b5xgb\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.193634 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.193602 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.193634 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.193614 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqws9\" (UniqueName: \"kubernetes.io/projected/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-kube-api-access-vqws9\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.193634 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.193628 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.193634 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.193640 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd4fcc7-38eb-40ff-b389-8b1fce29d132-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:28.270317 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.270278 4860 generic.go:356] "Generic (PLEG): container finished" podID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerID="e5d7b58377638c9c2714dfa776c4a2700db884b20e1de81751eb49b3f52d9bfd" exitCode=0 Nov 02 18:13:28.270502 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.270333 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" Nov 02 18:13:28.270502 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.270363 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" event={"ID":"ebd4fcc7-38eb-40ff-b389-8b1fce29d132","Type":"ContainerDied","Data":"e5d7b58377638c9c2714dfa776c4a2700db884b20e1de81751eb49b3f52d9bfd"} Nov 02 18:13:28.270502 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.270415 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d" event={"ID":"ebd4fcc7-38eb-40ff-b389-8b1fce29d132","Type":"ContainerDied","Data":"86dfaf21f5945982124aeb1617dc2ab93ce7d6f2e997aaf09ee38982243bfaf6"} Nov 02 18:13:28.270502 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.270439 4860 scope.go:122] "RemoveContainer" containerID="e5d7b58377638c9c2714dfa776c4a2700db884b20e1de81751eb49b3f52d9bfd" Nov 02 18:13:28.271933 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.271902 4860 generic.go:356] "Generic (PLEG): container finished" podID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerID="a65c79f3e530878efb797ed99d60140f0c380257673787b01bb59cc13afb0228" exitCode=0 Nov 02 18:13:28.272050 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.271978 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" Nov 02 18:13:28.272050 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.272005 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" event={"ID":"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5","Type":"ContainerDied","Data":"a65c79f3e530878efb797ed99d60140f0c380257673787b01bb59cc13afb0228"} Nov 02 18:13:28.272050 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.272040 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg" event={"ID":"12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5","Type":"ContainerDied","Data":"07724339e84c540c8a2a8f8b07af3700096040897b6a9e301b3c24d0b90f9905"} Nov 02 18:13:28.281393 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.281360 4860 scope.go:122] "RemoveContainer" containerID="e5d7b58377638c9c2714dfa776c4a2700db884b20e1de81751eb49b3f52d9bfd" Nov 02 18:13:28.281774 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:13:28.281703 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5d7b58377638c9c2714dfa776c4a2700db884b20e1de81751eb49b3f52d9bfd\": container with ID starting with e5d7b58377638c9c2714dfa776c4a2700db884b20e1de81751eb49b3f52d9bfd not found: ID does not exist" containerID="e5d7b58377638c9c2714dfa776c4a2700db884b20e1de81751eb49b3f52d9bfd" Nov 02 18:13:28.281774 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.281737 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5d7b58377638c9c2714dfa776c4a2700db884b20e1de81751eb49b3f52d9bfd"} err="failed to get container status \"e5d7b58377638c9c2714dfa776c4a2700db884b20e1de81751eb49b3f52d9bfd\": rpc error: code = NotFound desc = could not find container \"e5d7b58377638c9c2714dfa776c4a2700db884b20e1de81751eb49b3f52d9bfd\": container with ID starting with e5d7b58377638c9c2714dfa776c4a2700db884b20e1de81751eb49b3f52d9bfd not found: ID does not exist" Nov 02 18:13:28.281774 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.281760 4860 scope.go:122] "RemoveContainer" containerID="a65c79f3e530878efb797ed99d60140f0c380257673787b01bb59cc13afb0228" Nov 02 18:13:28.291813 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.291796 4860 scope.go:122] "RemoveContainer" containerID="a65c79f3e530878efb797ed99d60140f0c380257673787b01bb59cc13afb0228" Nov 02 18:13:28.292182 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:13:28.292149 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a65c79f3e530878efb797ed99d60140f0c380257673787b01bb59cc13afb0228\": container with ID starting with a65c79f3e530878efb797ed99d60140f0c380257673787b01bb59cc13afb0228 not found: ID does not exist" containerID="a65c79f3e530878efb797ed99d60140f0c380257673787b01bb59cc13afb0228" Nov 02 18:13:28.292468 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.292175 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65c79f3e530878efb797ed99d60140f0c380257673787b01bb59cc13afb0228"} err="failed to get container status \"a65c79f3e530878efb797ed99d60140f0c380257673787b01bb59cc13afb0228\": rpc error: code = NotFound desc = could not find container \"a65c79f3e530878efb797ed99d60140f0c380257673787b01bb59cc13afb0228\": container with ID starting with a65c79f3e530878efb797ed99d60140f0c380257673787b01bb59cc13afb0228 not found: ID does not exist" Nov 02 18:13:28.324162 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.324125 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d"] Nov 02 18:13:28.330144 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.330111 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/simulator-w6-7df526-kserve-76c8fc647f-qx96d"] Nov 02 18:13:28.364449 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.364417 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg"] Nov 02 18:13:28.371725 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:28.371665 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7df526/distinct-w6-7df526-kserve-544d955d49-9whlg"] Nov 02 18:13:29.499321 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:29.499288 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" path="/var/lib/kubelet/pods/12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5/volumes" Nov 02 18:13:29.499970 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:29.499948 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" path="/var/lib/kubelet/pods/ebd4fcc7-38eb-40ff-b389-8b1fce29d132/volumes" Nov 02 18:13:30.029703 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.029665 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" podUID="79d179ee-224e-4375-8cf1-863a774e1cb6" containerName="payload-processing" containerID="cri-o://5f03e926258629d0b4e3feec69fbaeb084795547f69ae4c0da0fd9344149fae6" gracePeriod=25 Nov 02 18:13:30.060919 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.060872 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" podUID="840ac652-3ceb-43ef-91b3-42c19179fe01" containerName="payload-pre-processing" containerID="cri-o://2b63690d74f4c643dd8a309039390ef233f26f0965fc14d8608ea3a86235f084" gracePeriod=25 Nov 02 18:13:30.161838 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.161789 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" containerID="cri-o://c2894fa0d6b5c8c09177b91e0b1f67cfda42a12ce15f90904d7361df59b9c80a" gracePeriod=45 Nov 02 18:13:30.291544 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.291511 4860 generic.go:356] "Generic (PLEG): container finished" podID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerID="c2894fa0d6b5c8c09177b91e0b1f67cfda42a12ce15f90904d7361df59b9c80a" exitCode=0 Nov 02 18:13:30.291700 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.291581 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" event={"ID":"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388","Type":"ContainerDied","Data":"c2894fa0d6b5c8c09177b91e0b1f67cfda42a12ce15f90904d7361df59b9c80a"} Nov 02 18:13:30.442516 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.442486 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 18:13:30.623381 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.623344 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-tmp-dir\") pod \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " Nov 02 18:13:30.624045 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.623442 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-kube-api-access-8mkch\" (UniqueName: \"kubernetes.io/projected/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-kube-api-access-8mkch\") pod \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " Nov 02 18:13:30.624045 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.623496 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-tls-certs\") pod \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " Nov 02 18:13:30.624045 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.623543 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-dshm\") pod \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " Nov 02 18:13:30.624045 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.623595 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-home\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-home\") pod \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " Nov 02 18:13:30.624045 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.623640 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-tmp-dir" pod "e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" (UID: "e7df1cfb-533d-4c2b-ad3c-1dbec34b0388"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:30.624045 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.623645 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-model-cache\") pod \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\" (UID: \"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388\") " Nov 02 18:13:30.624045 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.623897 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-home" pod "e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" (UID: "e7df1cfb-533d-4c2b-ad3c-1dbec34b0388"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:30.624045 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.623917 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-model-cache" pod "e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" (UID: "e7df1cfb-533d-4c2b-ad3c-1dbec34b0388"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:30.624540 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.624199 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:30.624540 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.624217 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:30.624540 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.624226 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:30.625944 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.625919 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-dshm" pod "e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" (UID: "e7df1cfb-533d-4c2b-ad3c-1dbec34b0388"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:30.626160 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.626137 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-tls-certs" pod "e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" (UID: "e7df1cfb-533d-4c2b-ad3c-1dbec34b0388"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:30.626208 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.626153 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-kube-api-access-8mkch" pod "e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" (UID: "e7df1cfb-533d-4c2b-ad3c-1dbec34b0388"). InnerVolumeSpecName "kube-api-access-8mkch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:30.725104 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.725069 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mkch\" (UniqueName: \"kubernetes.io/projected/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-kube-api-access-8mkch\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:30.725104 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.725099 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:30.725104 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:30.725109 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:31.299732 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:31.299704 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" Nov 02 18:13:31.299930 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:31.299799 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6" event={"ID":"e7df1cfb-533d-4c2b-ad3c-1dbec34b0388","Type":"ContainerDied","Data":"f8574df84943aeda43f97474bb29ea12877aed954e9954a706659dd24371e202"} Nov 02 18:13:31.299930 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:31.299869 4860 scope.go:122] "RemoveContainer" containerID="c2894fa0d6b5c8c09177b91e0b1f67cfda42a12ce15f90904d7361df59b9c80a" Nov 02 18:13:31.353365 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:31.353326 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6"] Nov 02 18:13:31.365599 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:31.365570 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/distinct-2-w3-5eedab-kserve-79bfd6c5d5-lrhj6"] Nov 02 18:13:31.499869 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:31.499826 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" path="/var/lib/kubelet/pods/e7df1cfb-533d-4c2b-ad3c-1dbec34b0388/volumes" Nov 02 18:13:32.220381 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.220344 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" containerID="cri-o://01401ae9f88c12f436dd91b4f3b25280bc4247ef59e053aa5c69d0dc4a1f7877" gracePeriod=45 Nov 02 18:13:32.234384 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.234343 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" containerID="cri-o://37073502e6da2fd11871626b4293fca85bf9b77558882532ccef5c0702c1b573" gracePeriod=45 Nov 02 18:13:32.234545 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.234325 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" containerID="cri-o://27d395bb387cc590929f34fa7a11e8adcd0345d8ec44c3d3c2a7a5410fd77099" gracePeriod=45 Nov 02 18:13:32.250392 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.250348 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" containerID="cri-o://a34c27d1d85194b32548c9b34ab349e9dfa062f7b1e69dded2140a551d123bf0" gracePeriod=45 Nov 02 18:13:32.566419 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.566385 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 18:13:32.624524 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.624495 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 18:13:32.628218 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.628192 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 18:13:32.632633 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.632606 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 18:13:32.748668 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.748632 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-dshm\") pod \"e50a1a79-1669-4ad8-89bb-d462a0e80100\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " Nov 02 18:13:32.748896 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.748681 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-model-cache\") pod \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " Nov 02 18:13:32.748896 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.748710 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-tmp-dir\") pod \"ce731d39-0cbe-4322-aa80-1d2606c3970a\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " Nov 02 18:13:32.748896 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.748739 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce731d39-0cbe-4322-aa80-1d2606c3970a-kube-api-access-xxn5k\" (UniqueName: \"kubernetes.io/projected/ce731d39-0cbe-4322-aa80-1d2606c3970a-kube-api-access-xxn5k\") pod \"ce731d39-0cbe-4322-aa80-1d2606c3970a\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " Nov 02 18:13:32.748896 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.748765 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf880079-551f-4973-ab8b-4009619a4fe5-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf880079-551f-4973-ab8b-4009619a4fe5-tls-certs\") pod \"cf880079-551f-4973-ab8b-4009619a4fe5\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " Nov 02 18:13:32.748896 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.748827 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-tmp-dir\") pod \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " Nov 02 18:13:32.748896 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.748870 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-dshm\") pod \"cf880079-551f-4973-ab8b-4009619a4fe5\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " Nov 02 18:13:32.748896 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.748896 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-home\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-home\") pod \"ce731d39-0cbe-4322-aa80-1d2606c3970a\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " Nov 02 18:13:32.749495 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.748921 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-tmp-dir\") pod \"e50a1a79-1669-4ad8-89bb-d462a0e80100\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " Nov 02 18:13:32.749495 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.748956 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e50a1a79-1669-4ad8-89bb-d462a0e80100-tls-certs\" (UniqueName: \"kubernetes.io/secret/e50a1a79-1669-4ad8-89bb-d462a0e80100-tls-certs\") pod \"e50a1a79-1669-4ad8-89bb-d462a0e80100\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " Nov 02 18:13:32.749495 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.749014 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-model-cache\") pod \"cf880079-551f-4973-ab8b-4009619a4fe5\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " Nov 02 18:13:32.749495 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.749082 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce731d39-0cbe-4322-aa80-1d2606c3970a-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce731d39-0cbe-4322-aa80-1d2606c3970a-tls-certs\") pod \"ce731d39-0cbe-4322-aa80-1d2606c3970a\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " Nov 02 18:13:32.749495 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.749170 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-tmp-dir" pod "ce731d39-0cbe-4322-aa80-1d2606c3970a" (UID: "ce731d39-0cbe-4322-aa80-1d2606c3970a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:32.749495 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.749215 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-model-cache\") pod \"ce731d39-0cbe-4322-aa80-1d2606c3970a\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " Nov 02 18:13:32.749495 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.749273 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-kube-api-access-fmcvs\" (UniqueName: \"kubernetes.io/projected/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-kube-api-access-fmcvs\") pod \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " Nov 02 18:13:32.749495 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.749309 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-model-cache\") pod \"e50a1a79-1669-4ad8-89bb-d462a0e80100\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " Nov 02 18:13:32.749495 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.749376 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-tmp-dir\") pod \"cf880079-551f-4973-ab8b-4009619a4fe5\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " Nov 02 18:13:32.749495 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.749415 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-tls-certs\") pod \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " Nov 02 18:13:32.749495 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.749469 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-home\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-home\") pod \"e50a1a79-1669-4ad8-89bb-d462a0e80100\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " Nov 02 18:13:32.750144 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.749522 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-home\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-home\") pod \"cf880079-551f-4973-ab8b-4009619a4fe5\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " Nov 02 18:13:32.750144 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.749619 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e50a1a79-1669-4ad8-89bb-d462a0e80100-kube-api-access-wxhq5\" (UniqueName: \"kubernetes.io/projected/e50a1a79-1669-4ad8-89bb-d462a0e80100-kube-api-access-wxhq5\") pod \"e50a1a79-1669-4ad8-89bb-d462a0e80100\" (UID: \"e50a1a79-1669-4ad8-89bb-d462a0e80100\") " Nov 02 18:13:32.750144 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.749649 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-home\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-home\") pod \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " Nov 02 18:13:32.750144 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.749684 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-dshm\") pod \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\" (UID: \"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c\") " Nov 02 18:13:32.750144 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.749733 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-dshm\") pod \"ce731d39-0cbe-4322-aa80-1d2606c3970a\" (UID: \"ce731d39-0cbe-4322-aa80-1d2606c3970a\") " Nov 02 18:13:32.750144 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.749784 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf880079-551f-4973-ab8b-4009619a4fe5-kube-api-access-98kms\" (UniqueName: \"kubernetes.io/projected/cf880079-551f-4973-ab8b-4009619a4fe5-kube-api-access-98kms\") pod \"cf880079-551f-4973-ab8b-4009619a4fe5\" (UID: \"cf880079-551f-4973-ab8b-4009619a4fe5\") " Nov 02 18:13:32.750478 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.750352 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.750658 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.749520 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-model-cache" pod "0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" (UID: "0d64cc17-abaf-4e5e-bb0d-b2ed1338437c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:32.750658 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.749758 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-model-cache" pod "e50a1a79-1669-4ad8-89bb-d462a0e80100" (UID: "e50a1a79-1669-4ad8-89bb-d462a0e80100"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:32.750658 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.750346 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-home" pod "ce731d39-0cbe-4322-aa80-1d2606c3970a" (UID: "ce731d39-0cbe-4322-aa80-1d2606c3970a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:32.750658 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.750583 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-tmp-dir" pod "e50a1a79-1669-4ad8-89bb-d462a0e80100" (UID: "e50a1a79-1669-4ad8-89bb-d462a0e80100"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:32.750658 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.750493 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-model-cache" pod "ce731d39-0cbe-4322-aa80-1d2606c3970a" (UID: "ce731d39-0cbe-4322-aa80-1d2606c3970a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:32.751466 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.751085 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-tmp-dir" pod "0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" (UID: "0d64cc17-abaf-4e5e-bb0d-b2ed1338437c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:32.751466 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.751212 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-home" pod "cf880079-551f-4973-ab8b-4009619a4fe5" (UID: "cf880079-551f-4973-ab8b-4009619a4fe5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:32.751466 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.751277 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-model-cache" pod "cf880079-551f-4973-ab8b-4009619a4fe5" (UID: "cf880079-551f-4973-ab8b-4009619a4fe5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:32.751466 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.751420 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-tmp-dir" pod "cf880079-551f-4973-ab8b-4009619a4fe5" (UID: "cf880079-551f-4973-ab8b-4009619a4fe5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:32.751466 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.751466 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-home" pod "e50a1a79-1669-4ad8-89bb-d462a0e80100" (UID: "e50a1a79-1669-4ad8-89bb-d462a0e80100"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:32.752074 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.752046 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-home" pod "0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" (UID: "0d64cc17-abaf-4e5e-bb0d-b2ed1338437c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:32.755039 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.754996 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf880079-551f-4973-ab8b-4009619a4fe5-kube-api-access-98kms" pod "cf880079-551f-4973-ab8b-4009619a4fe5" (UID: "cf880079-551f-4973-ab8b-4009619a4fe5"). InnerVolumeSpecName "kube-api-access-98kms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:32.755235 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.755212 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-dshm" pod "e50a1a79-1669-4ad8-89bb-d462a0e80100" (UID: "e50a1a79-1669-4ad8-89bb-d462a0e80100"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:32.755368 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.755350 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e50a1a79-1669-4ad8-89bb-d462a0e80100-tls-certs" pod "e50a1a79-1669-4ad8-89bb-d462a0e80100" (UID: "e50a1a79-1669-4ad8-89bb-d462a0e80100"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:32.755590 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.755552 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce731d39-0cbe-4322-aa80-1d2606c3970a-tls-certs" pod "ce731d39-0cbe-4322-aa80-1d2606c3970a" (UID: "ce731d39-0cbe-4322-aa80-1d2606c3970a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:32.755674 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.755650 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-dshm" pod "cf880079-551f-4973-ab8b-4009619a4fe5" (UID: "cf880079-551f-4973-ab8b-4009619a4fe5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:32.755992 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.755959 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf880079-551f-4973-ab8b-4009619a4fe5-tls-certs" pod "cf880079-551f-4973-ab8b-4009619a4fe5" (UID: "cf880079-551f-4973-ab8b-4009619a4fe5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:32.756078 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.756025 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-dshm" pod "ce731d39-0cbe-4322-aa80-1d2606c3970a" (UID: "ce731d39-0cbe-4322-aa80-1d2606c3970a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:32.758212 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.758173 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-dshm" pod "0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" (UID: "0d64cc17-abaf-4e5e-bb0d-b2ed1338437c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:32.758717 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.758680 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-kube-api-access-fmcvs" pod "0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" (UID: "0d64cc17-abaf-4e5e-bb0d-b2ed1338437c"). InnerVolumeSpecName "kube-api-access-fmcvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:32.759645 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.758715 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce731d39-0cbe-4322-aa80-1d2606c3970a-kube-api-access-xxn5k" pod "ce731d39-0cbe-4322-aa80-1d2606c3970a" (UID: "ce731d39-0cbe-4322-aa80-1d2606c3970a"). InnerVolumeSpecName "kube-api-access-xxn5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:32.760029 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.760005 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e50a1a79-1669-4ad8-89bb-d462a0e80100-kube-api-access-wxhq5" pod "e50a1a79-1669-4ad8-89bb-d462a0e80100" (UID: "e50a1a79-1669-4ad8-89bb-d462a0e80100"). InnerVolumeSpecName "kube-api-access-wxhq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:32.761312 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.760825 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-tls-certs" pod "0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" (UID: "0d64cc17-abaf-4e5e-bb0d-b2ed1338437c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:32.851723 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851677 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851723 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851720 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851723 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851733 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851742 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851750 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851758 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxhq5\" (UniqueName: \"kubernetes.io/projected/e50a1a79-1669-4ad8-89bb-d462a0e80100-kube-api-access-wxhq5\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851769 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851778 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851785 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851793 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98kms\" (UniqueName: \"kubernetes.io/projected/cf880079-551f-4973-ab8b-4009619a4fe5-kube-api-access-98kms\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851800 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851810 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851819 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxn5k\" (UniqueName: \"kubernetes.io/projected/ce731d39-0cbe-4322-aa80-1d2606c3970a-kube-api-access-xxn5k\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851826 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf880079-551f-4973-ab8b-4009619a4fe5-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851834 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851843 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851886 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851898 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50a1a79-1669-4ad8-89bb-d462a0e80100-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851908 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e50a1a79-1669-4ad8-89bb-d462a0e80100-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851915 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf880079-551f-4973-ab8b-4009619a4fe5-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851922 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce731d39-0cbe-4322-aa80-1d2606c3970a-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851929 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce731d39-0cbe-4322-aa80-1d2606c3970a-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:32.851991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:32.851937 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmcvs\" (UniqueName: \"kubernetes.io/projected/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c-kube-api-access-fmcvs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:33.316984 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.316941 4860 generic.go:356] "Generic (PLEG): container finished" podID="cf880079-551f-4973-ab8b-4009619a4fe5" containerID="01401ae9f88c12f436dd91b4f3b25280bc4247ef59e053aa5c69d0dc4a1f7877" exitCode=0 Nov 02 18:13:33.317452 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.317000 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" Nov 02 18:13:33.317452 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.317015 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" event={"ID":"cf880079-551f-4973-ab8b-4009619a4fe5","Type":"ContainerDied","Data":"01401ae9f88c12f436dd91b4f3b25280bc4247ef59e053aa5c69d0dc4a1f7877"} Nov 02 18:13:33.317452 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.317059 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52" event={"ID":"cf880079-551f-4973-ab8b-4009619a4fe5","Type":"ContainerDied","Data":"8cff816b4ff9fdb77dd2c8989b7e9217173c77fc93ab7ed9c27d08832c87dfd0"} Nov 02 18:13:33.317452 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.317076 4860 scope.go:122] "RemoveContainer" containerID="01401ae9f88c12f436dd91b4f3b25280bc4247ef59e053aa5c69d0dc4a1f7877" Nov 02 18:13:33.318665 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.318640 4860 generic.go:356] "Generic (PLEG): container finished" podID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerID="a34c27d1d85194b32548c9b34ab349e9dfa062f7b1e69dded2140a551d123bf0" exitCode=0 Nov 02 18:13:33.318875 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.318678 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" event={"ID":"ce731d39-0cbe-4322-aa80-1d2606c3970a","Type":"ContainerDied","Data":"a34c27d1d85194b32548c9b34ab349e9dfa062f7b1e69dded2140a551d123bf0"} Nov 02 18:13:33.318977 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.318713 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" Nov 02 18:13:33.318977 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.318900 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq" event={"ID":"ce731d39-0cbe-4322-aa80-1d2606c3970a","Type":"ContainerDied","Data":"af2a38f7e8c604398076bc64c42dd9600e8d2e7b02eb8f19c04e8ec7366f4a8c"} Nov 02 18:13:33.320554 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.320525 4860 generic.go:356] "Generic (PLEG): container finished" podID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerID="27d395bb387cc590929f34fa7a11e8adcd0345d8ec44c3d3c2a7a5410fd77099" exitCode=0 Nov 02 18:13:33.320687 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.320572 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" event={"ID":"e50a1a79-1669-4ad8-89bb-d462a0e80100","Type":"ContainerDied","Data":"27d395bb387cc590929f34fa7a11e8adcd0345d8ec44c3d3c2a7a5410fd77099"} Nov 02 18:13:33.320687 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.320610 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" event={"ID":"e50a1a79-1669-4ad8-89bb-d462a0e80100","Type":"ContainerDied","Data":"32faafefc61dac647c3b01b2e09719625ec7c08b9516b9ff6668cd8b14dbac93"} Nov 02 18:13:33.320687 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.320579 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf" Nov 02 18:13:33.322254 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.322235 4860 generic.go:356] "Generic (PLEG): container finished" podID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerID="37073502e6da2fd11871626b4293fca85bf9b77558882532ccef5c0702c1b573" exitCode=0 Nov 02 18:13:33.322357 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.322307 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" event={"ID":"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c","Type":"ContainerDied","Data":"37073502e6da2fd11871626b4293fca85bf9b77558882532ccef5c0702c1b573"} Nov 02 18:13:33.322357 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.322332 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" event={"ID":"0d64cc17-abaf-4e5e-bb0d-b2ed1338437c","Type":"ContainerDied","Data":"e6ecd99f57c5ce2a83c72eceb421ecd76f9f78b4d6709e96c91ab06ff33bcbf5"} Nov 02 18:13:33.322493 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.322380 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" Nov 02 18:13:33.331410 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.331388 4860 scope.go:122] "RemoveContainer" containerID="01401ae9f88c12f436dd91b4f3b25280bc4247ef59e053aa5c69d0dc4a1f7877" Nov 02 18:13:33.331890 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:13:33.331839 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01401ae9f88c12f436dd91b4f3b25280bc4247ef59e053aa5c69d0dc4a1f7877\": container with ID starting with 01401ae9f88c12f436dd91b4f3b25280bc4247ef59e053aa5c69d0dc4a1f7877 not found: ID does not exist" containerID="01401ae9f88c12f436dd91b4f3b25280bc4247ef59e053aa5c69d0dc4a1f7877" Nov 02 18:13:33.331982 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.331901 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01401ae9f88c12f436dd91b4f3b25280bc4247ef59e053aa5c69d0dc4a1f7877"} err="failed to get container status \"01401ae9f88c12f436dd91b4f3b25280bc4247ef59e053aa5c69d0dc4a1f7877\": rpc error: code = NotFound desc = could not find container \"01401ae9f88c12f436dd91b4f3b25280bc4247ef59e053aa5c69d0dc4a1f7877\": container with ID starting with 01401ae9f88c12f436dd91b4f3b25280bc4247ef59e053aa5c69d0dc4a1f7877 not found: ID does not exist" Nov 02 18:13:33.331982 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.331924 4860 scope.go:122] "RemoveContainer" containerID="a34c27d1d85194b32548c9b34ab349e9dfa062f7b1e69dded2140a551d123bf0" Nov 02 18:13:33.343318 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.343297 4860 scope.go:122] "RemoveContainer" containerID="a34c27d1d85194b32548c9b34ab349e9dfa062f7b1e69dded2140a551d123bf0" Nov 02 18:13:33.343664 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:13:33.343640 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a34c27d1d85194b32548c9b34ab349e9dfa062f7b1e69dded2140a551d123bf0\": container with ID starting with a34c27d1d85194b32548c9b34ab349e9dfa062f7b1e69dded2140a551d123bf0 not found: ID does not exist" containerID="a34c27d1d85194b32548c9b34ab349e9dfa062f7b1e69dded2140a551d123bf0" Nov 02 18:13:33.343750 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.343677 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34c27d1d85194b32548c9b34ab349e9dfa062f7b1e69dded2140a551d123bf0"} err="failed to get container status \"a34c27d1d85194b32548c9b34ab349e9dfa062f7b1e69dded2140a551d123bf0\": rpc error: code = NotFound desc = could not find container \"a34c27d1d85194b32548c9b34ab349e9dfa062f7b1e69dded2140a551d123bf0\": container with ID starting with a34c27d1d85194b32548c9b34ab349e9dfa062f7b1e69dded2140a551d123bf0 not found: ID does not exist" Nov 02 18:13:33.343750 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.343704 4860 scope.go:122] "RemoveContainer" containerID="27d395bb387cc590929f34fa7a11e8adcd0345d8ec44c3d3c2a7a5410fd77099" Nov 02 18:13:33.358198 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.358038 4860 scope.go:122] "RemoveContainer" containerID="27d395bb387cc590929f34fa7a11e8adcd0345d8ec44c3d3c2a7a5410fd77099" Nov 02 18:13:33.358440 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:13:33.358412 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d395bb387cc590929f34fa7a11e8adcd0345d8ec44c3d3c2a7a5410fd77099\": container with ID starting with 27d395bb387cc590929f34fa7a11e8adcd0345d8ec44c3d3c2a7a5410fd77099 not found: ID does not exist" containerID="27d395bb387cc590929f34fa7a11e8adcd0345d8ec44c3d3c2a7a5410fd77099" Nov 02 18:13:33.358540 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.358445 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d395bb387cc590929f34fa7a11e8adcd0345d8ec44c3d3c2a7a5410fd77099"} err="failed to get container status \"27d395bb387cc590929f34fa7a11e8adcd0345d8ec44c3d3c2a7a5410fd77099\": rpc error: code = NotFound desc = could not find container \"27d395bb387cc590929f34fa7a11e8adcd0345d8ec44c3d3c2a7a5410fd77099\": container with ID starting with 27d395bb387cc590929f34fa7a11e8adcd0345d8ec44c3d3c2a7a5410fd77099 not found: ID does not exist" Nov 02 18:13:33.358540 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.358464 4860 scope.go:122] "RemoveContainer" containerID="37073502e6da2fd11871626b4293fca85bf9b77558882532ccef5c0702c1b573" Nov 02 18:13:33.369595 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.369553 4860 scope.go:122] "RemoveContainer" containerID="37073502e6da2fd11871626b4293fca85bf9b77558882532ccef5c0702c1b573" Nov 02 18:13:33.369943 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:13:33.369916 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37073502e6da2fd11871626b4293fca85bf9b77558882532ccef5c0702c1b573\": container with ID starting with 37073502e6da2fd11871626b4293fca85bf9b77558882532ccef5c0702c1b573 not found: ID does not exist" containerID="37073502e6da2fd11871626b4293fca85bf9b77558882532ccef5c0702c1b573" Nov 02 18:13:33.370069 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.369950 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37073502e6da2fd11871626b4293fca85bf9b77558882532ccef5c0702c1b573"} err="failed to get container status \"37073502e6da2fd11871626b4293fca85bf9b77558882532ccef5c0702c1b573\": rpc error: code = NotFound desc = could not find container \"37073502e6da2fd11871626b4293fca85bf9b77558882532ccef5c0702c1b573\": container with ID starting with 37073502e6da2fd11871626b4293fca85bf9b77558882532ccef5c0702c1b573 not found: ID does not exist" Nov 02 18:13:33.374996 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.374968 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52"] Nov 02 18:13:33.386915 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.386894 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/distinct-2-w4-2899ab-kserve-77c8984777-plj52"] Nov 02 18:13:33.431233 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.431205 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq"] Nov 02 18:13:33.434425 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.434395 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/simulator-w4-2899ab-kserve-5b68dfc6b6-l7fsq"] Nov 02 18:13:33.485280 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.485244 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg"] Nov 02 18:13:33.490075 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.490010 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg"] Nov 02 18:13:33.501300 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.501268 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" path="/var/lib/kubelet/pods/0d64cc17-abaf-4e5e-bb0d-b2ed1338437c/volumes" Nov 02 18:13:33.501784 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.501768 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" path="/var/lib/kubelet/pods/ce731d39-0cbe-4322-aa80-1d2606c3970a/volumes" Nov 02 18:13:33.502215 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.502201 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" path="/var/lib/kubelet/pods/cf880079-551f-4973-ab8b-4009619a4fe5/volumes" Nov 02 18:13:33.528236 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.528204 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf"] Nov 02 18:13:33.534616 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.534588 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2899ab/premium-w4-2899ab-kserve-5d6f649c8c-mmwpf"] Nov 02 18:13:33.617341 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:33.617301 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2899ab/distinct-w4-2899ab-kserve-d67dd5bf8-9rsvg" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:13:34.349901 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.349726 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" containerID="cri-o://d43bc000ddb3cf0e3e21882f2466225022fe58942b8ce5c2e7168baf4e36fbc3" gracePeriod=45 Nov 02 18:13:34.661018 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.660992 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 18:13:34.776774 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.776715 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/355a0269-1933-46ee-b701-5a5c45da4061-tls-certs\" (UniqueName: \"kubernetes.io/secret/355a0269-1933-46ee-b701-5a5c45da4061-tls-certs\") pod \"355a0269-1933-46ee-b701-5a5c45da4061\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " Nov 02 18:13:34.776774 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.776756 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-dshm\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-dshm\") pod \"355a0269-1933-46ee-b701-5a5c45da4061\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " Nov 02 18:13:34.777066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.776821 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-home\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-home\") pod \"355a0269-1933-46ee-b701-5a5c45da4061\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " Nov 02 18:13:34.777066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.776875 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-tmp-dir\") pod \"355a0269-1933-46ee-b701-5a5c45da4061\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " Nov 02 18:13:34.777066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.776901 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355a0269-1933-46ee-b701-5a5c45da4061-kube-api-access-wq9f5\" (UniqueName: \"kubernetes.io/projected/355a0269-1933-46ee-b701-5a5c45da4061-kube-api-access-wq9f5\") pod \"355a0269-1933-46ee-b701-5a5c45da4061\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " Nov 02 18:13:34.777066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.776937 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-model-cache\") pod \"355a0269-1933-46ee-b701-5a5c45da4061\" (UID: \"355a0269-1933-46ee-b701-5a5c45da4061\") " Nov 02 18:13:34.777618 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.777585 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-model-cache" pod "355a0269-1933-46ee-b701-5a5c45da4061" (UID: "355a0269-1933-46ee-b701-5a5c45da4061"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:34.777813 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.777789 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-tmp-dir" pod "355a0269-1933-46ee-b701-5a5c45da4061" (UID: "355a0269-1933-46ee-b701-5a5c45da4061"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:34.777927 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.777597 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-home" pod "355a0269-1933-46ee-b701-5a5c45da4061" (UID: "355a0269-1933-46ee-b701-5a5c45da4061"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:34.780293 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.780268 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-dshm" pod "355a0269-1933-46ee-b701-5a5c45da4061" (UID: "355a0269-1933-46ee-b701-5a5c45da4061"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:34.780420 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.780401 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/355a0269-1933-46ee-b701-5a5c45da4061-tls-certs" pod "355a0269-1933-46ee-b701-5a5c45da4061" (UID: "355a0269-1933-46ee-b701-5a5c45da4061"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:34.780959 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.780939 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355a0269-1933-46ee-b701-5a5c45da4061-kube-api-access-wq9f5" pod "355a0269-1933-46ee-b701-5a5c45da4061" (UID: "355a0269-1933-46ee-b701-5a5c45da4061"). InnerVolumeSpecName "kube-api-access-wq9f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:34.879051 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.878648 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/355a0269-1933-46ee-b701-5a5c45da4061-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:34.879051 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.878690 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:34.879051 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.878704 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:34.879051 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.878719 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:34.879051 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.878734 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq9f5\" (UniqueName: \"kubernetes.io/projected/355a0269-1933-46ee-b701-5a5c45da4061-kube-api-access-wq9f5\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:34.879051 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:34.878749 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/355a0269-1933-46ee-b701-5a5c45da4061-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:35.345373 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:35.345336 4860 generic.go:356] "Generic (PLEG): container finished" podID="355a0269-1933-46ee-b701-5a5c45da4061" containerID="d43bc000ddb3cf0e3e21882f2466225022fe58942b8ce5c2e7168baf4e36fbc3" exitCode=0 Nov 02 18:13:35.345556 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:35.345385 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" Nov 02 18:13:35.345556 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:35.345406 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" event={"ID":"355a0269-1933-46ee-b701-5a5c45da4061","Type":"ContainerDied","Data":"d43bc000ddb3cf0e3e21882f2466225022fe58942b8ce5c2e7168baf4e36fbc3"} Nov 02 18:13:35.345556 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:35.345438 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79" event={"ID":"355a0269-1933-46ee-b701-5a5c45da4061","Type":"ContainerDied","Data":"f331f78513996eb0d17ac6da039162ef5665db066c8a069fa553963140749d09"} Nov 02 18:13:35.345556 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:35.345454 4860 scope.go:122] "RemoveContainer" containerID="d43bc000ddb3cf0e3e21882f2466225022fe58942b8ce5c2e7168baf4e36fbc3" Nov 02 18:13:35.356767 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:35.356743 4860 scope.go:122] "RemoveContainer" containerID="d43bc000ddb3cf0e3e21882f2466225022fe58942b8ce5c2e7168baf4e36fbc3" Nov 02 18:13:35.357108 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:13:35.357067 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d43bc000ddb3cf0e3e21882f2466225022fe58942b8ce5c2e7168baf4e36fbc3\": container with ID starting with d43bc000ddb3cf0e3e21882f2466225022fe58942b8ce5c2e7168baf4e36fbc3 not found: ID does not exist" containerID="d43bc000ddb3cf0e3e21882f2466225022fe58942b8ce5c2e7168baf4e36fbc3" Nov 02 18:13:35.357108 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:35.357095 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d43bc000ddb3cf0e3e21882f2466225022fe58942b8ce5c2e7168baf4e36fbc3"} err="failed to get container status \"d43bc000ddb3cf0e3e21882f2466225022fe58942b8ce5c2e7168baf4e36fbc3\": rpc error: code = NotFound desc = could not find container \"d43bc000ddb3cf0e3e21882f2466225022fe58942b8ce5c2e7168baf4e36fbc3\": container with ID starting with d43bc000ddb3cf0e3e21882f2466225022fe58942b8ce5c2e7168baf4e36fbc3 not found: ID does not exist" Nov 02 18:13:35.397537 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:35.397503 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79"] Nov 02 18:13:35.409639 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:35.409581 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/distinct-w5-863b87-kserve-77c7998f8b-x8p79"] Nov 02 18:13:35.499422 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:35.499383 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="355a0269-1933-46ee-b701-5a5c45da4061" path="/var/lib/kubelet/pods/355a0269-1933-46ee-b701-5a5c45da4061/volumes" Nov 02 18:13:35.499835 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:35.499818 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" path="/var/lib/kubelet/pods/e50a1a79-1669-4ad8-89bb-d462a0e80100/volumes" Nov 02 18:13:37.291941 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:37.291900 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" podUID="a27864d8-7810-45a3-8cf1-ccc0033b4268" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:13:38.298014 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:38.297976 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" podUID="79d179ee-224e-4375-8cf1-863a774e1cb6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:13:39.305552 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:39.305512 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" podUID="840ac652-3ceb-43ef-91b3-42c19179fe01" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:13:45.612006 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:45.611937 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" Nov 02 18:13:45.690415 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:45.690369 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a27864d8-7810-45a3-8cf1-ccc0033b4268-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27864d8-7810-45a3-8cf1-ccc0033b4268-plugins-config-volume\") pod \"a27864d8-7810-45a3-8cf1-ccc0033b4268\" (UID: \"a27864d8-7810-45a3-8cf1-ccc0033b4268\") " Nov 02 18:13:45.690575 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:45.690548 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a27864d8-7810-45a3-8cf1-ccc0033b4268-kube-api-access-f8d5z\" (UniqueName: \"kubernetes.io/projected/a27864d8-7810-45a3-8cf1-ccc0033b4268-kube-api-access-f8d5z\") pod \"a27864d8-7810-45a3-8cf1-ccc0033b4268\" (UID: \"a27864d8-7810-45a3-8cf1-ccc0033b4268\") " Nov 02 18:13:45.690825 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:45.690799 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a27864d8-7810-45a3-8cf1-ccc0033b4268-plugins-config-volume" pod "a27864d8-7810-45a3-8cf1-ccc0033b4268" (UID: "a27864d8-7810-45a3-8cf1-ccc0033b4268"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:45.690955 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:45.690933 4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27864d8-7810-45a3-8cf1-ccc0033b4268-plugins-config-volume\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:45.692735 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:45.692711 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27864d8-7810-45a3-8cf1-ccc0033b4268-kube-api-access-f8d5z" pod "a27864d8-7810-45a3-8cf1-ccc0033b4268" (UID: "a27864d8-7810-45a3-8cf1-ccc0033b4268"). InnerVolumeSpecName "kube-api-access-f8d5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:45.792294 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:45.792248 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8d5z\" (UniqueName: \"kubernetes.io/projected/a27864d8-7810-45a3-8cf1-ccc0033b4268-kube-api-access-f8d5z\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:46.423108 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:46.423068 4860 generic.go:356] "Generic (PLEG): container finished" podID="a27864d8-7810-45a3-8cf1-ccc0033b4268" containerID="56e35e030b02e0bbeb93edf23cbd27ee008616642485520516f30425cff477b1" exitCode=0 Nov 02 18:13:46.423108 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:46.423109 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" Nov 02 18:13:46.423345 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:46.423142 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" event={"ID":"a27864d8-7810-45a3-8cf1-ccc0033b4268","Type":"ContainerDied","Data":"56e35e030b02e0bbeb93edf23cbd27ee008616642485520516f30425cff477b1"} Nov 02 18:13:46.423345 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:46.423194 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd" event={"ID":"a27864d8-7810-45a3-8cf1-ccc0033b4268","Type":"ContainerDied","Data":"585becf0a15ca959b002fb5d2f11b1a52461cd059f0b220482e3983083942c27"} Nov 02 18:13:46.423345 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:46.423215 4860 scope.go:122] "RemoveContainer" containerID="56e35e030b02e0bbeb93edf23cbd27ee008616642485520516f30425cff477b1" Nov 02 18:13:46.433471 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:46.433446 4860 scope.go:122] "RemoveContainer" containerID="56e35e030b02e0bbeb93edf23cbd27ee008616642485520516f30425cff477b1" Nov 02 18:13:46.434061 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:13:46.434033 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e35e030b02e0bbeb93edf23cbd27ee008616642485520516f30425cff477b1\": container with ID starting with 56e35e030b02e0bbeb93edf23cbd27ee008616642485520516f30425cff477b1 not found: ID does not exist" containerID="56e35e030b02e0bbeb93edf23cbd27ee008616642485520516f30425cff477b1" Nov 02 18:13:46.434167 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:46.434069 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e35e030b02e0bbeb93edf23cbd27ee008616642485520516f30425cff477b1"} err="failed to get container status \"56e35e030b02e0bbeb93edf23cbd27ee008616642485520516f30425cff477b1\": rpc error: code = NotFound desc = could not find container \"56e35e030b02e0bbeb93edf23cbd27ee008616642485520516f30425cff477b1\": container with ID starting with 56e35e030b02e0bbeb93edf23cbd27ee008616642485520516f30425cff477b1 not found: ID does not exist" Nov 02 18:13:46.475878 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:46.475830 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd"] Nov 02 18:13:46.488203 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:46.488167 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5eedab-758bdcc969-9jqbd"] Nov 02 18:13:47.498334 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:47.498295 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a27864d8-7810-45a3-8cf1-ccc0033b4268" path="/var/lib/kubelet/pods/a27864d8-7810-45a3-8cf1-ccc0033b4268/volumes" Nov 02 18:13:48.297316 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:48.297270 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" podUID="79d179ee-224e-4375-8cf1-863a774e1cb6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:13:49.332896 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:49.321327 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" podUID="840ac652-3ceb-43ef-91b3-42c19179fe01" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:13:50.336929 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.336903 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" Nov 02 18:13:50.372767 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.372742 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" Nov 02 18:13:50.442982 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.442949 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d179ee-224e-4375-8cf1-863a774e1cb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d179ee-224e-4375-8cf1-863a774e1cb6-plugins-config-volume\") pod \"79d179ee-224e-4375-8cf1-863a774e1cb6\" (UID: \"79d179ee-224e-4375-8cf1-863a774e1cb6\") " Nov 02 18:13:50.443159 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.443120 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/840ac652-3ceb-43ef-91b3-42c19179fe01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/840ac652-3ceb-43ef-91b3-42c19179fe01-plugins-config-volume\") pod \"840ac652-3ceb-43ef-91b3-42c19179fe01\" (UID: \"840ac652-3ceb-43ef-91b3-42c19179fe01\") " Nov 02 18:13:50.443159 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.443147 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/840ac652-3ceb-43ef-91b3-42c19179fe01-kube-api-access-s2mwk\" (UniqueName: \"kubernetes.io/projected/840ac652-3ceb-43ef-91b3-42c19179fe01-kube-api-access-s2mwk\") pod \"840ac652-3ceb-43ef-91b3-42c19179fe01\" (UID: \"840ac652-3ceb-43ef-91b3-42c19179fe01\") " Nov 02 18:13:50.443260 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.443207 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d179ee-224e-4375-8cf1-863a774e1cb6-kube-api-access-dnw6b\" (UniqueName: \"kubernetes.io/projected/79d179ee-224e-4375-8cf1-863a774e1cb6-kube-api-access-dnw6b\") pod \"79d179ee-224e-4375-8cf1-863a774e1cb6\" (UID: \"79d179ee-224e-4375-8cf1-863a774e1cb6\") " Nov 02 18:13:50.443548 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.443517 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d179ee-224e-4375-8cf1-863a774e1cb6-plugins-config-volume" pod "79d179ee-224e-4375-8cf1-863a774e1cb6" (UID: "79d179ee-224e-4375-8cf1-863a774e1cb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:50.443675 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.443550 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/840ac652-3ceb-43ef-91b3-42c19179fe01-plugins-config-volume" pod "840ac652-3ceb-43ef-91b3-42c19179fe01" (UID: "840ac652-3ceb-43ef-91b3-42c19179fe01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:50.445437 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.445408 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/840ac652-3ceb-43ef-91b3-42c19179fe01-kube-api-access-s2mwk" pod "840ac652-3ceb-43ef-91b3-42c19179fe01" (UID: "840ac652-3ceb-43ef-91b3-42c19179fe01"). InnerVolumeSpecName "kube-api-access-s2mwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:50.445546 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.445454 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d179ee-224e-4375-8cf1-863a774e1cb6-kube-api-access-dnw6b" pod "79d179ee-224e-4375-8cf1-863a774e1cb6" (UID: "79d179ee-224e-4375-8cf1-863a774e1cb6"). InnerVolumeSpecName "kube-api-access-dnw6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:50.452630 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.452607 4860 generic.go:356] "Generic (PLEG): container finished" podID="840ac652-3ceb-43ef-91b3-42c19179fe01" containerID="2b63690d74f4c643dd8a309039390ef233f26f0965fc14d8608ea3a86235f084" exitCode=0 Nov 02 18:13:50.452737 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.452666 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" Nov 02 18:13:50.452737 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.452698 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" event={"ID":"840ac652-3ceb-43ef-91b3-42c19179fe01","Type":"ContainerDied","Data":"2b63690d74f4c643dd8a309039390ef233f26f0965fc14d8608ea3a86235f084"} Nov 02 18:13:50.452825 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.452768 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t" event={"ID":"840ac652-3ceb-43ef-91b3-42c19179fe01","Type":"ContainerDied","Data":"27b66f7dbec561f481457e5d5827bba170838266133e50d796ab9bb7176af262"} Nov 02 18:13:50.452825 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.452803 4860 scope.go:122] "RemoveContainer" containerID="2b63690d74f4c643dd8a309039390ef233f26f0965fc14d8608ea3a86235f084" Nov 02 18:13:50.454438 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.454415 4860 generic.go:356] "Generic (PLEG): container finished" podID="79d179ee-224e-4375-8cf1-863a774e1cb6" containerID="5f03e926258629d0b4e3feec69fbaeb084795547f69ae4c0da0fd9344149fae6" exitCode=0 Nov 02 18:13:50.454561 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.454458 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" Nov 02 18:13:50.454561 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.454486 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" event={"ID":"79d179ee-224e-4375-8cf1-863a774e1cb6","Type":"ContainerDied","Data":"5f03e926258629d0b4e3feec69fbaeb084795547f69ae4c0da0fd9344149fae6"} Nov 02 18:13:50.454561 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.454523 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr" event={"ID":"79d179ee-224e-4375-8cf1-863a774e1cb6","Type":"ContainerDied","Data":"6f4461a7a8ce32fb9a58a522fa49cb2d28817d366d80f6afcac7d659b2803b21"} Nov 02 18:13:50.464219 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.464198 4860 scope.go:122] "RemoveContainer" containerID="2b63690d74f4c643dd8a309039390ef233f26f0965fc14d8608ea3a86235f084" Nov 02 18:13:50.464522 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:13:50.464499 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b63690d74f4c643dd8a309039390ef233f26f0965fc14d8608ea3a86235f084\": container with ID starting with 2b63690d74f4c643dd8a309039390ef233f26f0965fc14d8608ea3a86235f084 not found: ID does not exist" containerID="2b63690d74f4c643dd8a309039390ef233f26f0965fc14d8608ea3a86235f084" Nov 02 18:13:50.464621 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.464527 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b63690d74f4c643dd8a309039390ef233f26f0965fc14d8608ea3a86235f084"} err="failed to get container status \"2b63690d74f4c643dd8a309039390ef233f26f0965fc14d8608ea3a86235f084\": rpc error: code = NotFound desc = could not find container \"2b63690d74f4c643dd8a309039390ef233f26f0965fc14d8608ea3a86235f084\": container with ID starting with 2b63690d74f4c643dd8a309039390ef233f26f0965fc14d8608ea3a86235f084 not found: ID does not exist" Nov 02 18:13:50.464621 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.464544 4860 scope.go:122] "RemoveContainer" containerID="5f03e926258629d0b4e3feec69fbaeb084795547f69ae4c0da0fd9344149fae6" Nov 02 18:13:50.474068 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.474046 4860 scope.go:122] "RemoveContainer" containerID="5f03e926258629d0b4e3feec69fbaeb084795547f69ae4c0da0fd9344149fae6" Nov 02 18:13:50.474329 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:13:50.474307 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f03e926258629d0b4e3feec69fbaeb084795547f69ae4c0da0fd9344149fae6\": container with ID starting with 5f03e926258629d0b4e3feec69fbaeb084795547f69ae4c0da0fd9344149fae6 not found: ID does not exist" containerID="5f03e926258629d0b4e3feec69fbaeb084795547f69ae4c0da0fd9344149fae6" Nov 02 18:13:50.474384 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.474338 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f03e926258629d0b4e3feec69fbaeb084795547f69ae4c0da0fd9344149fae6"} err="failed to get container status \"5f03e926258629d0b4e3feec69fbaeb084795547f69ae4c0da0fd9344149fae6\": rpc error: code = NotFound desc = could not find container \"5f03e926258629d0b4e3feec69fbaeb084795547f69ae4c0da0fd9344149fae6\": container with ID starting with 5f03e926258629d0b4e3feec69fbaeb084795547f69ae4c0da0fd9344149fae6 not found: ID does not exist" Nov 02 18:13:50.509592 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.509565 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t"] Nov 02 18:13:50.512541 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.512516 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-863b87-5b7564c947-kjx2t"] Nov 02 18:13:50.544794 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.544708 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnw6b\" (UniqueName: \"kubernetes.io/projected/79d179ee-224e-4375-8cf1-863a774e1cb6-kube-api-access-dnw6b\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:50.544794 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.544745 4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d179ee-224e-4375-8cf1-863a774e1cb6-plugins-config-volume\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:50.544794 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.544760 4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/840ac652-3ceb-43ef-91b3-42c19179fe01-plugins-config-volume\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:50.544794 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.544773 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2mwk\" (UniqueName: \"kubernetes.io/projected/840ac652-3ceb-43ef-91b3-42c19179fe01-kube-api-access-s2mwk\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:50.549586 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.549547 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr"] Nov 02 18:13:50.554990 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:50.554960 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-863b87-5ffdd78b65-6sqgr"] Nov 02 18:13:51.500085 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:51.500043 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d179ee-224e-4375-8cf1-863a774e1cb6" path="/var/lib/kubelet/pods/79d179ee-224e-4375-8cf1-863a774e1cb6/volumes" Nov 02 18:13:51.500652 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:51.500631 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="840ac652-3ceb-43ef-91b3-42c19179fe01" path="/var/lib/kubelet/pods/840ac652-3ceb-43ef-91b3-42c19179fe01/volumes" Nov 02 18:13:51.773721 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:13:51.773638 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:13:51.798885 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:51.798816 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4"] Nov 02 18:13:51.799101 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:51.799054 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" podUID="e0839624-4c7c-42fe-acc1-7515cea20996" containerName="maas-api" containerID="cri-o://771bacacc944ca343be59572a5feae7f535fc0d411f9fb6075da32179e97ca61" gracePeriod=30 Nov 02 18:13:52.122897 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.121439 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" Nov 02 18:13:52.181306 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.181274 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0839624-4c7c-42fe-acc1-7515cea20996-kube-api-access-2pbtp\" (UniqueName: \"kubernetes.io/projected/e0839624-4c7c-42fe-acc1-7515cea20996-kube-api-access-2pbtp\") pod \"e0839624-4c7c-42fe-acc1-7515cea20996\" (UID: \"e0839624-4c7c-42fe-acc1-7515cea20996\") " Nov 02 18:13:52.181464 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.181329 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0839624-4c7c-42fe-acc1-7515cea20996-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0839624-4c7c-42fe-acc1-7515cea20996-maas-api-tls\") pod \"e0839624-4c7c-42fe-acc1-7515cea20996\" (UID: \"e0839624-4c7c-42fe-acc1-7515cea20996\") " Nov 02 18:13:52.181464 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.181410 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0839624-4c7c-42fe-acc1-7515cea20996-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0839624-4c7c-42fe-acc1-7515cea20996-metrics-certs\") pod \"e0839624-4c7c-42fe-acc1-7515cea20996\" (UID: \"e0839624-4c7c-42fe-acc1-7515cea20996\") " Nov 02 18:13:52.181464 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.181452 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0839624-4c7c-42fe-acc1-7515cea20996-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0839624-4c7c-42fe-acc1-7515cea20996-openshift-service-ca\") pod \"e0839624-4c7c-42fe-acc1-7515cea20996\" (UID: \"e0839624-4c7c-42fe-acc1-7515cea20996\") " Nov 02 18:13:52.182161 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.182132 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0839624-4c7c-42fe-acc1-7515cea20996-openshift-service-ca" pod "e0839624-4c7c-42fe-acc1-7515cea20996" (UID: "e0839624-4c7c-42fe-acc1-7515cea20996"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:52.184051 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.184019 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0839624-4c7c-42fe-acc1-7515cea20996-metrics-certs" pod "e0839624-4c7c-42fe-acc1-7515cea20996" (UID: "e0839624-4c7c-42fe-acc1-7515cea20996"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:52.184168 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.184121 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0839624-4c7c-42fe-acc1-7515cea20996-kube-api-access-2pbtp" pod "e0839624-4c7c-42fe-acc1-7515cea20996" (UID: "e0839624-4c7c-42fe-acc1-7515cea20996"). InnerVolumeSpecName "kube-api-access-2pbtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:52.184223 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.184192 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0839624-4c7c-42fe-acc1-7515cea20996-maas-api-tls" pod "e0839624-4c7c-42fe-acc1-7515cea20996" (UID: "e0839624-4c7c-42fe-acc1-7515cea20996"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:52.283364 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.283326 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0839624-4c7c-42fe-acc1-7515cea20996-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:52.283364 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.283356 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0839624-4c7c-42fe-acc1-7515cea20996-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:52.283364 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.283367 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pbtp\" (UniqueName: \"kubernetes.io/projected/e0839624-4c7c-42fe-acc1-7515cea20996-kube-api-access-2pbtp\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:52.283364 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.283377 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0839624-4c7c-42fe-acc1-7515cea20996-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:13:52.475057 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.475010 4860 generic.go:356] "Generic (PLEG): container finished" podID="e0839624-4c7c-42fe-acc1-7515cea20996" containerID="771bacacc944ca343be59572a5feae7f535fc0d411f9fb6075da32179e97ca61" exitCode=0 Nov 02 18:13:52.475228 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.475081 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" Nov 02 18:13:52.475228 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.475101 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" event={"ID":"e0839624-4c7c-42fe-acc1-7515cea20996","Type":"ContainerDied","Data":"771bacacc944ca343be59572a5feae7f535fc0d411f9fb6075da32179e97ca61"} Nov 02 18:13:52.475228 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.475157 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4" event={"ID":"e0839624-4c7c-42fe-acc1-7515cea20996","Type":"ContainerDied","Data":"d3562ae869dc6c9d565bc7176ef4282935d047c482c3ffefa30fdfaec1629c75"} Nov 02 18:13:52.475228 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.475180 4860 scope.go:122] "RemoveContainer" containerID="771bacacc944ca343be59572a5feae7f535fc0d411f9fb6075da32179e97ca61" Nov 02 18:13:52.489842 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.489643 4860 scope.go:122] "RemoveContainer" containerID="771bacacc944ca343be59572a5feae7f535fc0d411f9fb6075da32179e97ca61" Nov 02 18:13:52.490205 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:13:52.490178 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"771bacacc944ca343be59572a5feae7f535fc0d411f9fb6075da32179e97ca61\": container with ID starting with 771bacacc944ca343be59572a5feae7f535fc0d411f9fb6075da32179e97ca61 not found: ID does not exist" containerID="771bacacc944ca343be59572a5feae7f535fc0d411f9fb6075da32179e97ca61" Nov 02 18:13:52.490303 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.490213 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"771bacacc944ca343be59572a5feae7f535fc0d411f9fb6075da32179e97ca61"} err="failed to get container status \"771bacacc944ca343be59572a5feae7f535fc0d411f9fb6075da32179e97ca61\": rpc error: code = NotFound desc = could not find container \"771bacacc944ca343be59572a5feae7f535fc0d411f9fb6075da32179e97ca61\": container with ID starting with 771bacacc944ca343be59572a5feae7f535fc0d411f9fb6075da32179e97ca61 not found: ID does not exist" Nov 02 18:13:52.529630 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.529597 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4"] Nov 02 18:13:52.541534 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:52.541442 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-863b87-5886444974-qm9z4"] Nov 02 18:13:53.498334 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:13:53.498240 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0839624-4c7c-42fe-acc1-7515cea20996" path="/var/lib/kubelet/pods/e0839624-4c7c-42fe-acc1-7515cea20996/volumes" Nov 02 18:14:00.267490 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:00.267447 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm"] Nov 02 18:14:00.268344 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:00.268107 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" podUID="a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" containerName="istio-proxy" containerID="cri-o://b6e188e7917a333690af41ed1f45f82985e8f79e89d4104ea56412bd9bb1937a" gracePeriod=30 Nov 02 18:14:01.054871 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:01.054084 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6"] Nov 02 18:14:01.054871 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:01.054339 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" podUID="df8332d1-5646-4263-957d-58fa676e8775" containerName="istio-proxy" containerID="cri-o://23a0702d9b64b96ffbd9117bd626910e7693754b970d9ef1a66b6126884c7d15" gracePeriod=30 Nov 02 18:14:03.167990 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:03.167947 4860 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:14:03.168547 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:03.168027 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" podUID="df8332d1-5646-4263-957d-58fa676e8775" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:14:05.543758 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.543698 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 18:14:05.588217 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.588182 4860 generic.go:356] "Generic (PLEG): container finished" podID="a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" containerID="b6e188e7917a333690af41ed1f45f82985e8f79e89d4104ea56412bd9bb1937a" exitCode=0 Nov 02 18:14:05.588365 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.588243 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" Nov 02 18:14:05.588365 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.588248 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" event={"ID":"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4","Type":"ContainerDied","Data":"b6e188e7917a333690af41ed1f45f82985e8f79e89d4104ea56412bd9bb1937a"} Nov 02 18:14:05.588365 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.588315 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm" event={"ID":"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4","Type":"ContainerDied","Data":"67e2237cf2b1cce58f1b4f80120b677040928ddb1a94582876b9f0bed69858d3"} Nov 02 18:14:05.588365 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.588348 4860 scope.go:122] "RemoveContainer" containerID="b6e188e7917a333690af41ed1f45f82985e8f79e89d4104ea56412bd9bb1937a" Nov 02 18:14:05.601728 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.601697 4860 scope.go:122] "RemoveContainer" containerID="b6e188e7917a333690af41ed1f45f82985e8f79e89d4104ea56412bd9bb1937a" Nov 02 18:14:05.602118 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:14:05.602091 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e188e7917a333690af41ed1f45f82985e8f79e89d4104ea56412bd9bb1937a\": container with ID starting with b6e188e7917a333690af41ed1f45f82985e8f79e89d4104ea56412bd9bb1937a not found: ID does not exist" containerID="b6e188e7917a333690af41ed1f45f82985e8f79e89d4104ea56412bd9bb1937a" Nov 02 18:14:05.602222 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.602131 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e188e7917a333690af41ed1f45f82985e8f79e89d4104ea56412bd9bb1937a"} err="failed to get container status \"b6e188e7917a333690af41ed1f45f82985e8f79e89d4104ea56412bd9bb1937a\": rpc error: code = NotFound desc = could not find container \"b6e188e7917a333690af41ed1f45f82985e8f79e89d4104ea56412bd9bb1937a\": container with ID starting with b6e188e7917a333690af41ed1f45f82985e8f79e89d4104ea56412bd9bb1937a not found: ID does not exist" Nov 02 18:14:05.635134 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.635099 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-workload-socket\") pod \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " Nov 02 18:14:05.635289 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.635153 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-kube-api-access-7wpls\" (UniqueName: \"kubernetes.io/projected/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-kube-api-access-7wpls\") pod \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " Nov 02 18:14:05.635289 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.635200 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-workload-certs\") pod \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " Nov 02 18:14:05.635289 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.635239 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-envoy\") pod \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " Nov 02 18:14:05.635289 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.635276 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-podinfo\") pod \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " Nov 02 18:14:05.635507 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.635320 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istiod-ca-cert\") pod \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " Nov 02 18:14:05.635507 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.635377 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-token\" (UniqueName: \"kubernetes.io/projected/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-token\") pod \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " Nov 02 18:14:05.635507 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.635404 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-workload-socket" pod "a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" (UID: "a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:14:05.635507 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.635428 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-data\") pod \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " Nov 02 18:14:05.635634 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.635518 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-credential-socket\") pod \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\" (UID: \"a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4\") " Nov 02 18:14:05.635804 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.635774 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-workload-certs" pod "a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" (UID: "a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:14:05.636103 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.635819 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:14:05.636232 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.636091 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istiod-ca-cert" pod "a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" (UID: "a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:14:05.636232 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.636195 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-data" pod "a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" (UID: "a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:14:05.636404 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.636248 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-credential-socket" pod "a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" (UID: "a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:14:05.637813 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.637770 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-kube-api-access-7wpls" pod "a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" (UID: "a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4"). InnerVolumeSpecName "kube-api-access-7wpls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:14:05.637979 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.637954 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-podinfo" pod "a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" (UID: "a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:14:05.638058 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.638033 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-envoy" pod "a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" (UID: "a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:14:05.638118 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.638075 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-token" pod "a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" (UID: "a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:14:05.737187 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.737153 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:14:05.737187 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.737183 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wpls\" (UniqueName: \"kubernetes.io/projected/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-kube-api-access-7wpls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:14:05.737187 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.737194 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:14:05.737427 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.737203 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:14:05.737427 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.737211 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:14:05.737427 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.737219 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:14:05.737427 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.737227 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:14:05.737427 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.737236 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:14:05.935636 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.935591 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm"] Nov 02 18:14:05.947944 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:05.947909 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-6q7tm"] Nov 02 18:14:06.290560 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.290532 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 18:14:06.342603 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.342565 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-workload-socket\") pod \"df8332d1-5646-4263-957d-58fa676e8775\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " Nov 02 18:14:06.342790 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.342635 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df8332d1-5646-4263-957d-58fa676e8775-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8332d1-5646-4263-957d-58fa676e8775-istiod-ca-cert\") pod \"df8332d1-5646-4263-957d-58fa676e8775\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " Nov 02 18:14:06.342790 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.342670 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df8332d1-5646-4263-957d-58fa676e8775-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8332d1-5646-4263-957d-58fa676e8775-istio-podinfo\") pod \"df8332d1-5646-4263-957d-58fa676e8775\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " Nov 02 18:14:06.342790 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.342700 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-istio-envoy\") pod \"df8332d1-5646-4263-957d-58fa676e8775\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " Nov 02 18:14:06.342790 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.342747 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8332d1-5646-4263-957d-58fa676e8775-istio-token\" (UniqueName: \"kubernetes.io/projected/df8332d1-5646-4263-957d-58fa676e8775-istio-token\") pod \"df8332d1-5646-4263-957d-58fa676e8775\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " Nov 02 18:14:06.342790 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.342790 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-workload-certs\") pod \"df8332d1-5646-4263-957d-58fa676e8775\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " Nov 02 18:14:06.343110 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.342836 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8332d1-5646-4263-957d-58fa676e8775-kube-api-access-kmbwd\" (UniqueName: \"kubernetes.io/projected/df8332d1-5646-4263-957d-58fa676e8775-kube-api-access-kmbwd\") pod \"df8332d1-5646-4263-957d-58fa676e8775\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " Nov 02 18:14:06.343110 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.342893 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-workload-socket" pod "df8332d1-5646-4263-957d-58fa676e8775" (UID: "df8332d1-5646-4263-957d-58fa676e8775"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:14:06.343110 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.342901 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-istio-data\") pod \"df8332d1-5646-4263-957d-58fa676e8775\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " Nov 02 18:14:06.343110 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.342990 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-credential-socket\") pod \"df8332d1-5646-4263-957d-58fa676e8775\" (UID: \"df8332d1-5646-4263-957d-58fa676e8775\") " Nov 02 18:14:06.343319 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.343120 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df8332d1-5646-4263-957d-58fa676e8775-istiod-ca-cert" pod "df8332d1-5646-4263-957d-58fa676e8775" (UID: "df8332d1-5646-4263-957d-58fa676e8775"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:14:06.343319 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.343139 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-workload-certs" pod "df8332d1-5646-4263-957d-58fa676e8775" (UID: "df8332d1-5646-4263-957d-58fa676e8775"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:14:06.343479 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.343403 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-credential-socket" pod "df8332d1-5646-4263-957d-58fa676e8775" (UID: "df8332d1-5646-4263-957d-58fa676e8775"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:14:06.343535 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.343466 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:14:06.343535 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.343499 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8332d1-5646-4263-957d-58fa676e8775-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:14:06.343535 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.343513 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:14:06.343678 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.343611 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-istio-data" pod "df8332d1-5646-4263-957d-58fa676e8775" (UID: "df8332d1-5646-4263-957d-58fa676e8775"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:14:06.345344 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.345314 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df8332d1-5646-4263-957d-58fa676e8775-istio-podinfo" pod "df8332d1-5646-4263-957d-58fa676e8775" (UID: "df8332d1-5646-4263-957d-58fa676e8775"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:14:06.345556 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.345530 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8332d1-5646-4263-957d-58fa676e8775-kube-api-access-kmbwd" pod "df8332d1-5646-4263-957d-58fa676e8775" (UID: "df8332d1-5646-4263-957d-58fa676e8775"). InnerVolumeSpecName "kube-api-access-kmbwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:14:06.345778 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.345756 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-istio-envoy" pod "df8332d1-5646-4263-957d-58fa676e8775" (UID: "df8332d1-5646-4263-957d-58fa676e8775"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:14:06.345907 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.345888 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8332d1-5646-4263-957d-58fa676e8775-istio-token" pod "df8332d1-5646-4263-957d-58fa676e8775" (UID: "df8332d1-5646-4263-957d-58fa676e8775"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:14:06.444105 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.444027 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmbwd\" (UniqueName: \"kubernetes.io/projected/df8332d1-5646-4263-957d-58fa676e8775-kube-api-access-kmbwd\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:14:06.444105 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.444057 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:14:06.444105 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.444066 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:14:06.444105 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.444076 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8332d1-5646-4263-957d-58fa676e8775-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:14:06.444105 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.444085 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8332d1-5646-4263-957d-58fa676e8775-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:14:06.444105 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.444092 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8332d1-5646-4263-957d-58fa676e8775-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:14:06.597167 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.597133 4860 generic.go:356] "Generic (PLEG): container finished" podID="df8332d1-5646-4263-957d-58fa676e8775" containerID="23a0702d9b64b96ffbd9117bd626910e7693754b970d9ef1a66b6126884c7d15" exitCode=0 Nov 02 18:14:06.597592 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.597185 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" Nov 02 18:14:06.597592 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.597227 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" event={"ID":"df8332d1-5646-4263-957d-58fa676e8775","Type":"ContainerDied","Data":"23a0702d9b64b96ffbd9117bd626910e7693754b970d9ef1a66b6126884c7d15"} Nov 02 18:14:06.597592 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.597282 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6" event={"ID":"df8332d1-5646-4263-957d-58fa676e8775","Type":"ContainerDied","Data":"1ff20032378e2b02db63aadc1eea1809771c7534228510bf5464c78af3810791"} Nov 02 18:14:06.597592 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.597305 4860 scope.go:122] "RemoveContainer" containerID="23a0702d9b64b96ffbd9117bd626910e7693754b970d9ef1a66b6126884c7d15" Nov 02 18:14:06.608006 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.607988 4860 scope.go:122] "RemoveContainer" containerID="23a0702d9b64b96ffbd9117bd626910e7693754b970d9ef1a66b6126884c7d15" Nov 02 18:14:06.608276 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:14:06.608259 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23a0702d9b64b96ffbd9117bd626910e7693754b970d9ef1a66b6126884c7d15\": container with ID starting with 23a0702d9b64b96ffbd9117bd626910e7693754b970d9ef1a66b6126884c7d15 not found: ID does not exist" containerID="23a0702d9b64b96ffbd9117bd626910e7693754b970d9ef1a66b6126884c7d15" Nov 02 18:14:06.608323 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.608284 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23a0702d9b64b96ffbd9117bd626910e7693754b970d9ef1a66b6126884c7d15"} err="failed to get container status \"23a0702d9b64b96ffbd9117bd626910e7693754b970d9ef1a66b6126884c7d15\": rpc error: code = NotFound desc = could not find container \"23a0702d9b64b96ffbd9117bd626910e7693754b970d9ef1a66b6126884c7d15\": container with ID starting with 23a0702d9b64b96ffbd9117bd626910e7693754b970d9ef1a66b6126884c7d15 not found: ID does not exist" Nov 02 18:14:06.650303 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.650272 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6"] Nov 02 18:14:06.662176 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:06.662145 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-l87f6"] Nov 02 18:14:07.499597 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:07.499561 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" path="/var/lib/kubelet/pods/a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4/volumes" Nov 02 18:14:07.500054 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:07.500038 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df8332d1-5646-4263-957d-58fa676e8775" path="/var/lib/kubelet/pods/df8332d1-5646-4263-957d-58fa676e8775/volumes" Nov 02 18:14:21.769718 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:14:21.769678 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:14:42.099190 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:42.099160 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 18:14:42.111735 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:42.111711 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 18:14:42.112103 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:42.112080 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 18:14:42.123862 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:14:42.123825 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 18:14:51.770487 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:14:51.770444 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:15:10.925558 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:10.925517 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs"] Nov 02 18:15:10.926198 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:10.925785 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" podUID="134da1f2-4c24-495f-9b0a-76d21ba04df4" containerName="maas-api" containerID="cri-o://5749a8cd039c6ecdfa34638533d303da9163cabb5dca7c79c3b44a11e40fcd31" gracePeriod=30 Nov 02 18:15:11.205646 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:11.205575 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" Nov 02 18:15:11.317690 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:11.317654 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134da1f2-4c24-495f-9b0a-76d21ba04df4-kube-api-access-5tjv5\" (UniqueName: \"kubernetes.io/projected/134da1f2-4c24-495f-9b0a-76d21ba04df4-kube-api-access-5tjv5\") pod \"134da1f2-4c24-495f-9b0a-76d21ba04df4\" (UID: \"134da1f2-4c24-495f-9b0a-76d21ba04df4\") " Nov 02 18:15:11.317895 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:11.317702 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/134da1f2-4c24-495f-9b0a-76d21ba04df4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/134da1f2-4c24-495f-9b0a-76d21ba04df4-openshift-service-ca\") pod \"134da1f2-4c24-495f-9b0a-76d21ba04df4\" (UID: \"134da1f2-4c24-495f-9b0a-76d21ba04df4\") " Nov 02 18:15:11.317947 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:11.317917 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/134da1f2-4c24-495f-9b0a-76d21ba04df4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/134da1f2-4c24-495f-9b0a-76d21ba04df4-maas-api-tls\") pod \"134da1f2-4c24-495f-9b0a-76d21ba04df4\" (UID: \"134da1f2-4c24-495f-9b0a-76d21ba04df4\") " Nov 02 18:15:11.317988 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:11.317951 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/134da1f2-4c24-495f-9b0a-76d21ba04df4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/134da1f2-4c24-495f-9b0a-76d21ba04df4-metrics-certs\") pod \"134da1f2-4c24-495f-9b0a-76d21ba04df4\" (UID: \"134da1f2-4c24-495f-9b0a-76d21ba04df4\") " Nov 02 18:15:11.318204 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:11.318180 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134da1f2-4c24-495f-9b0a-76d21ba04df4-openshift-service-ca" pod "134da1f2-4c24-495f-9b0a-76d21ba04df4" (UID: "134da1f2-4c24-495f-9b0a-76d21ba04df4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:11.318322 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:11.318296 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/134da1f2-4c24-495f-9b0a-76d21ba04df4-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:15:11.320011 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:11.319979 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134da1f2-4c24-495f-9b0a-76d21ba04df4-kube-api-access-5tjv5" pod "134da1f2-4c24-495f-9b0a-76d21ba04df4" (UID: "134da1f2-4c24-495f-9b0a-76d21ba04df4"). InnerVolumeSpecName "kube-api-access-5tjv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:11.320148 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:11.320024 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/134da1f2-4c24-495f-9b0a-76d21ba04df4-metrics-certs" pod "134da1f2-4c24-495f-9b0a-76d21ba04df4" (UID: "134da1f2-4c24-495f-9b0a-76d21ba04df4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:15:11.320227 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:11.320165 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/134da1f2-4c24-495f-9b0a-76d21ba04df4-maas-api-tls" pod "134da1f2-4c24-495f-9b0a-76d21ba04df4" (UID: "134da1f2-4c24-495f-9b0a-76d21ba04df4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:15:11.419768 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:11.419727 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/134da1f2-4c24-495f-9b0a-76d21ba04df4-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:15:11.419768 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:11.419760 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/134da1f2-4c24-495f-9b0a-76d21ba04df4-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:15:11.419768 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:11.419770 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tjv5\" (UniqueName: \"kubernetes.io/projected/134da1f2-4c24-495f-9b0a-76d21ba04df4-kube-api-access-5tjv5\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:15:12.027004 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:12.026969 4860 generic.go:356] "Generic (PLEG): container finished" podID="134da1f2-4c24-495f-9b0a-76d21ba04df4" containerID="5749a8cd039c6ecdfa34638533d303da9163cabb5dca7c79c3b44a11e40fcd31" exitCode=0 Nov 02 18:15:12.027510 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:12.027018 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" Nov 02 18:15:12.027510 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:12.027063 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" event={"ID":"134da1f2-4c24-495f-9b0a-76d21ba04df4","Type":"ContainerDied","Data":"5749a8cd039c6ecdfa34638533d303da9163cabb5dca7c79c3b44a11e40fcd31"} Nov 02 18:15:12.027510 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:12.027134 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs" event={"ID":"134da1f2-4c24-495f-9b0a-76d21ba04df4","Type":"ContainerDied","Data":"19c0b39c21858081f7ca89acdc091209edcfb1e0a6d204213cc8327e5726ba52"} Nov 02 18:15:12.027510 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:12.027170 4860 scope.go:122] "RemoveContainer" containerID="5749a8cd039c6ecdfa34638533d303da9163cabb5dca7c79c3b44a11e40fcd31" Nov 02 18:15:12.037576 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:12.037555 4860 scope.go:122] "RemoveContainer" containerID="5749a8cd039c6ecdfa34638533d303da9163cabb5dca7c79c3b44a11e40fcd31" Nov 02 18:15:12.037893 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:15:12.037872 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5749a8cd039c6ecdfa34638533d303da9163cabb5dca7c79c3b44a11e40fcd31\": container with ID starting with 5749a8cd039c6ecdfa34638533d303da9163cabb5dca7c79c3b44a11e40fcd31 not found: ID does not exist" containerID="5749a8cd039c6ecdfa34638533d303da9163cabb5dca7c79c3b44a11e40fcd31" Nov 02 18:15:12.037966 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:12.037900 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5749a8cd039c6ecdfa34638533d303da9163cabb5dca7c79c3b44a11e40fcd31"} err="failed to get container status \"5749a8cd039c6ecdfa34638533d303da9163cabb5dca7c79c3b44a11e40fcd31\": rpc error: code = NotFound desc = could not find container \"5749a8cd039c6ecdfa34638533d303da9163cabb5dca7c79c3b44a11e40fcd31\": container with ID starting with 5749a8cd039c6ecdfa34638533d303da9163cabb5dca7c79c3b44a11e40fcd31 not found: ID does not exist" Nov 02 18:15:12.080144 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:12.080116 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs"] Nov 02 18:15:12.092173 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:12.092149 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f91b2d-f458cbc5b-scpbs"] Nov 02 18:15:13.499807 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:13.499773 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="134da1f2-4c24-495f-9b0a-76d21ba04df4" path="/var/lib/kubelet/pods/134da1f2-4c24-495f-9b0a-76d21ba04df4/volumes" Nov 02 18:15:21.769572 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:15:21.769529 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:15:32.966644 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:32.966608 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t"] Nov 02 18:15:32.967234 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:32.966863 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" podUID="82dde012-8d96-4cc2-8887-99d8e9833f23" containerName="maas-api" containerID="cri-o://065ceb15eaee3960b17ebbbbe4a1aad30b044da2c95423818ecee1f3299983fb" gracePeriod=30 Nov 02 18:15:33.164741 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:33.164688 4860 generic.go:356] "Generic (PLEG): container finished" podID="82dde012-8d96-4cc2-8887-99d8e9833f23" containerID="065ceb15eaee3960b17ebbbbe4a1aad30b044da2c95423818ecee1f3299983fb" exitCode=0 Nov 02 18:15:33.165003 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:33.164768 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" event={"ID":"82dde012-8d96-4cc2-8887-99d8e9833f23","Type":"ContainerDied","Data":"065ceb15eaee3960b17ebbbbe4a1aad30b044da2c95423818ecee1f3299983fb"} Nov 02 18:15:33.240057 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:33.240031 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" Nov 02 18:15:33.344094 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:33.344057 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82dde012-8d96-4cc2-8887-99d8e9833f23-kube-api-access-cnzvz\" (UniqueName: \"kubernetes.io/projected/82dde012-8d96-4cc2-8887-99d8e9833f23-kube-api-access-cnzvz\") pod \"82dde012-8d96-4cc2-8887-99d8e9833f23\" (UID: \"82dde012-8d96-4cc2-8887-99d8e9833f23\") " Nov 02 18:15:33.344294 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:33.344125 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82dde012-8d96-4cc2-8887-99d8e9833f23-metrics-certs\" (UniqueName: \"kubernetes.io/secret/82dde012-8d96-4cc2-8887-99d8e9833f23-metrics-certs\") pod \"82dde012-8d96-4cc2-8887-99d8e9833f23\" (UID: \"82dde012-8d96-4cc2-8887-99d8e9833f23\") " Nov 02 18:15:33.344294 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:33.344232 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82dde012-8d96-4cc2-8887-99d8e9833f23-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82dde012-8d96-4cc2-8887-99d8e9833f23-maas-api-tls\") pod \"82dde012-8d96-4cc2-8887-99d8e9833f23\" (UID: \"82dde012-8d96-4cc2-8887-99d8e9833f23\") " Nov 02 18:15:33.344294 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:33.344268 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82dde012-8d96-4cc2-8887-99d8e9833f23-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82dde012-8d96-4cc2-8887-99d8e9833f23-openshift-service-ca\") pod \"82dde012-8d96-4cc2-8887-99d8e9833f23\" (UID: \"82dde012-8d96-4cc2-8887-99d8e9833f23\") " Nov 02 18:15:33.344685 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:33.344656 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82dde012-8d96-4cc2-8887-99d8e9833f23-openshift-service-ca" pod "82dde012-8d96-4cc2-8887-99d8e9833f23" (UID: "82dde012-8d96-4cc2-8887-99d8e9833f23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:33.346250 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:33.346217 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82dde012-8d96-4cc2-8887-99d8e9833f23-kube-api-access-cnzvz" pod "82dde012-8d96-4cc2-8887-99d8e9833f23" (UID: "82dde012-8d96-4cc2-8887-99d8e9833f23"). InnerVolumeSpecName "kube-api-access-cnzvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:33.346691 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:33.346662 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82dde012-8d96-4cc2-8887-99d8e9833f23-maas-api-tls" pod "82dde012-8d96-4cc2-8887-99d8e9833f23" (UID: "82dde012-8d96-4cc2-8887-99d8e9833f23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:15:33.346807 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:33.346674 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82dde012-8d96-4cc2-8887-99d8e9833f23-metrics-certs" pod "82dde012-8d96-4cc2-8887-99d8e9833f23" (UID: "82dde012-8d96-4cc2-8887-99d8e9833f23"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:15:33.445623 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:33.445592 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82dde012-8d96-4cc2-8887-99d8e9833f23-maas-api-tls\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:15:33.445623 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:33.445619 4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82dde012-8d96-4cc2-8887-99d8e9833f23-openshift-service-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:15:33.445623 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:33.445629 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnzvz\" (UniqueName: \"kubernetes.io/projected/82dde012-8d96-4cc2-8887-99d8e9833f23-kube-api-access-cnzvz\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:15:33.445981 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:33.445638 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82dde012-8d96-4cc2-8887-99d8e9833f23-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:15:34.173046 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:34.173008 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" event={"ID":"82dde012-8d96-4cc2-8887-99d8e9833f23","Type":"ContainerDied","Data":"08b33c11cd846e9cf0eebfc989c8552093c2abdda37caa252073e1d594500dc1"} Nov 02 18:15:34.173480 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:34.173054 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t" Nov 02 18:15:34.173480 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:34.173097 4860 scope.go:122] "RemoveContainer" containerID="065ceb15eaee3960b17ebbbbe4a1aad30b044da2c95423818ecee1f3299983fb" Nov 02 18:15:34.225544 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:34.225511 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t"] Nov 02 18:15:34.237768 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:34.237738 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c4dbe3-d9cf486d-bkm4t"] Nov 02 18:15:35.498081 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:35.498047 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82dde012-8d96-4cc2-8887-99d8e9833f23" path="/var/lib/kubelet/pods/82dde012-8d96-4cc2-8887-99d8e9833f23/volumes" Nov 02 18:15:46.617186 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:46.617149 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc"] Nov 02 18:15:46.629368 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:46.629337 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65"] Nov 02 18:15:46.639523 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:46.639495 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw"] Nov 02 18:15:51.769053 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:15:51.769010 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:15:52.384459 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:52.384423 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp"] Nov 02 18:15:52.430309 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:52.430272 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2"] Nov 02 18:15:54.931433 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.931389 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj"] Nov 02 18:15:54.932009 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.931988 4860 cpu_manager.go:414] "containerMap: removing container" podUID="82dde012-8d96-4cc2-8887-99d8e9833f23" containerName="maas-api" Nov 02 18:15:54.932009 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932010 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82dde012-8d96-4cc2-8887-99d8e9833f23" containerName="maas-api" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932022 4860 cpu_manager.go:414] "containerMap: removing container" podUID="e1e2db1f-bd9c-46f0-b913-9533fcf95a4d" containerName="payload-pre-processing" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932031 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e2db1f-bd9c-46f0-b913-9533fcf95a4d" containerName="payload-pre-processing" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932046 4860 cpu_manager.go:414] "containerMap: removing container" podUID="8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" containerName="istio-proxy" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932051 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" containerName="istio-proxy" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932062 4860 cpu_manager.go:414] "containerMap: removing container" podUID="1fb27095-a4ff-429a-91be-6f3e28872f18" containerName="istio-proxy" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932068 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb27095-a4ff-429a-91be-6f3e28872f18" containerName="istio-proxy" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932080 4860 cpu_manager.go:414] "containerMap: removing container" podUID="79d179ee-224e-4375-8cf1-863a774e1cb6" containerName="payload-processing" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932085 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d179ee-224e-4375-8cf1-863a774e1cb6" containerName="payload-processing" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932092 4860 cpu_manager.go:414] "containerMap: removing container" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932096 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932107 4860 cpu_manager.go:414] "containerMap: removing container" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932113 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932120 4860 cpu_manager.go:414] "containerMap: removing container" podUID="d4332a04-bc02-4fce-925b-df037762c76f" containerName="maas-api" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932126 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4332a04-bc02-4fce-925b-df037762c76f" containerName="maas-api" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932133 4860 cpu_manager.go:414] "containerMap: removing container" podUID="10a632f9-0197-4f8c-b414-c15b29caea6e" containerName="discovery" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932137 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a632f9-0197-4f8c-b414-c15b29caea6e" containerName="discovery" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932143 4860 cpu_manager.go:414] "containerMap: removing container" podUID="5b585147-38f8-4a8f-be0b-d1ffaef9481d" containerName="istio-proxy" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932148 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b585147-38f8-4a8f-be0b-d1ffaef9481d" containerName="istio-proxy" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932155 4860 cpu_manager.go:414] "containerMap: removing container" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932159 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932165 4860 cpu_manager.go:414] "containerMap: removing container" podUID="95ef1178-de19-4f53-9646-f4ac3ed52f57" containerName="istio-proxy" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932169 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ef1178-de19-4f53-9646-f4ac3ed52f57" containerName="istio-proxy" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932177 4860 cpu_manager.go:414] "containerMap: removing container" podUID="c92d9c87-7ba0-4d6e-b659-9e916730c79e" containerName="istio-proxy" Nov 02 18:15:54.932165 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932182 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c92d9c87-7ba0-4d6e-b659-9e916730c79e" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932188 4860 cpu_manager.go:414] "containerMap: removing container" podUID="e0839624-4c7c-42fe-acc1-7515cea20996" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932193 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0839624-4c7c-42fe-acc1-7515cea20996" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932201 4860 cpu_manager.go:414] "containerMap: removing container" podUID="62b05251-7074-41bc-8994-c1051abee475" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932208 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b05251-7074-41bc-8994-c1051abee475" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932220 4860 cpu_manager.go:414] "containerMap: removing container" podUID="a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932225 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932232 4860 cpu_manager.go:414] "containerMap: removing container" podUID="16e30f93-2270-4fb7-bfc2-3e48edf5ecc1" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932237 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e30f93-2270-4fb7-bfc2-3e48edf5ecc1" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932250 4860 cpu_manager.go:414] "containerMap: removing container" podUID="a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932256 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932270 4860 cpu_manager.go:414] "containerMap: removing container" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932277 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932287 4860 cpu_manager.go:414] "containerMap: removing container" podUID="151de229-e8e3-4bc3-8443-b12cfca9aab8" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932294 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151de229-e8e3-4bc3-8443-b12cfca9aab8" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932305 4860 cpu_manager.go:414] "containerMap: removing container" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932311 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932321 4860 cpu_manager.go:414] "containerMap: removing container" podUID="66504ddb-d869-4549-b870-61b42e58e871" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932328 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66504ddb-d869-4549-b870-61b42e58e871" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932338 4860 cpu_manager.go:414] "containerMap: removing container" podUID="e9293d64-b25e-4076-b707-fc65717da6a8" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932345 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9293d64-b25e-4076-b707-fc65717da6a8" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932353 4860 cpu_manager.go:414] "containerMap: removing container" podUID="cef174d1-f0e7-4748-997a-fa814937890c" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932360 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cef174d1-f0e7-4748-997a-fa814937890c" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932372 4860 cpu_manager.go:414] "containerMap: removing container" podUID="dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932379 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932390 4860 cpu_manager.go:414] "containerMap: removing container" podUID="a27864d8-7810-45a3-8cf1-ccc0033b4268" containerName="payload-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932398 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a27864d8-7810-45a3-8cf1-ccc0033b4268" containerName="payload-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932411 4860 cpu_manager.go:414] "containerMap: removing container" podUID="62b05251-7074-41bc-8994-c1051abee475" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932418 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b05251-7074-41bc-8994-c1051abee475" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932430 4860 cpu_manager.go:414] "containerMap: removing container" podUID="edbe40ed-0184-44e4-990c-2b7a3fd51f5f" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932437 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edbe40ed-0184-44e4-990c-2b7a3fd51f5f" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932447 4860 cpu_manager.go:414] "containerMap: removing container" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932455 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932467 4860 cpu_manager.go:414] "containerMap: removing container" podUID="b53fb7b2-d91c-409b-bf83-e24b95a7608f" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932474 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b53fb7b2-d91c-409b-bf83-e24b95a7608f" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932489 4860 cpu_manager.go:414] "containerMap: removing container" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932496 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932505 4860 cpu_manager.go:414] "containerMap: removing container" podUID="63743df5-0f1d-4708-8c41-80b2c9344fc5" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932512 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63743df5-0f1d-4708-8c41-80b2c9344fc5" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932527 4860 cpu_manager.go:414] "containerMap: removing container" podUID="39a03fdb-9060-4f2c-a96e-d78764568df8" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932533 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39a03fdb-9060-4f2c-a96e-d78764568df8" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932544 4860 cpu_manager.go:414] "containerMap: removing container" podUID="f95eaf1d-668d-405f-bbc1-6be67fbe1fed" containerName="payload-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932552 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95eaf1d-668d-405f-bbc1-6be67fbe1fed" containerName="payload-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932561 4860 cpu_manager.go:414] "containerMap: removing container" podUID="75e58f61-43a2-4600-a380-549625283343" containerName="payload-pre-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932568 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e58f61-43a2-4600-a380-549625283343" containerName="payload-pre-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932578 4860 cpu_manager.go:414] "containerMap: removing container" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932584 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932594 4860 cpu_manager.go:414] "containerMap: removing container" podUID="df8332d1-5646-4263-957d-58fa676e8775" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932602 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8332d1-5646-4263-957d-58fa676e8775" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932612 4860 cpu_manager.go:414] "containerMap: removing container" podUID="719f739a-813b-4f51-9bd4-e523bba953af" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932618 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="719f739a-813b-4f51-9bd4-e523bba953af" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932627 4860 cpu_manager.go:414] "containerMap: removing container" podUID="6277b932-8a69-4947-afe2-0946a7c6f177" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932634 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6277b932-8a69-4947-afe2-0946a7c6f177" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932645 4860 cpu_manager.go:414] "containerMap: removing container" podUID="bafb2593-6516-446c-b113-37bff6320729" containerName="payload-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932652 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bafb2593-6516-446c-b113-37bff6320729" containerName="payload-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932666 4860 cpu_manager.go:414] "containerMap: removing container" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932673 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932679 4860 cpu_manager.go:414] "containerMap: removing container" podUID="42544ef4-8181-4c6e-bbd8-337a63fc0412" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932684 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42544ef4-8181-4c6e-bbd8-337a63fc0412" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932692 4860 cpu_manager.go:414] "containerMap: removing container" podUID="caee9964-de70-4471-9013-90b120d20fa1" containerName="payload-pre-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932698 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caee9964-de70-4471-9013-90b120d20fa1" containerName="payload-pre-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932705 4860 cpu_manager.go:414] "containerMap: removing container" podUID="9a9aa9e4-1f8b-4512-ae25-51b759bd1c91" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932710 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a9aa9e4-1f8b-4512-ae25-51b759bd1c91" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932716 4860 cpu_manager.go:414] "containerMap: removing container" podUID="69187cd1-8708-46d3-91d9-62b9a9446454" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932721 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69187cd1-8708-46d3-91d9-62b9a9446454" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932729 4860 cpu_manager.go:414] "containerMap: removing container" podUID="d7e0682c-4519-452a-ab21-9781064a6260" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932733 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e0682c-4519-452a-ab21-9781064a6260" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932739 4860 cpu_manager.go:414] "containerMap: removing container" podUID="134da1f2-4c24-495f-9b0a-76d21ba04df4" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932744 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="134da1f2-4c24-495f-9b0a-76d21ba04df4" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932751 4860 cpu_manager.go:414] "containerMap: removing container" podUID="71b9f322-2bf8-415b-a26a-3723a34c2482" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932756 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b9f322-2bf8-415b-a26a-3723a34c2482" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932765 4860 cpu_manager.go:414] "containerMap: removing container" podUID="ceba04e6-9968-4301-aaa3-6372edea5720" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932770 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceba04e6-9968-4301-aaa3-6372edea5720" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932776 4860 cpu_manager.go:414] "containerMap: removing container" podUID="63359d54-2671-4cbc-9033-cc1716d65580" containerName="payload-pre-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932781 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63359d54-2671-4cbc-9033-cc1716d65580" containerName="payload-pre-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932787 4860 cpu_manager.go:414] "containerMap: removing container" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932791 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932797 4860 cpu_manager.go:414] "containerMap: removing container" podUID="980e42a5-a602-46ea-842d-6b6914e1b870" containerName="payload-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932801 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="980e42a5-a602-46ea-842d-6b6914e1b870" containerName="payload-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932807 4860 cpu_manager.go:414] "containerMap: removing container" podUID="840ac652-3ceb-43ef-91b3-42c19179fe01" containerName="payload-pre-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932812 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="840ac652-3ceb-43ef-91b3-42c19179fe01" containerName="payload-pre-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932820 4860 cpu_manager.go:414] "containerMap: removing container" podUID="76282517-6c00-4e83-b5f5-a6cfaa0dcad2" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932825 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76282517-6c00-4e83-b5f5-a6cfaa0dcad2" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932831 4860 cpu_manager.go:414] "containerMap: removing container" podUID="c12481ab-3025-40e3-a9a7-50b44df24d70" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932835 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c12481ab-3025-40e3-a9a7-50b44df24d70" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932843 4860 cpu_manager.go:414] "containerMap: removing container" podUID="6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932865 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932876 4860 cpu_manager.go:414] "containerMap: removing container" podUID="e31cdb01-9f71-4eae-ad26-6f1e19915ffd" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932883 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e31cdb01-9f71-4eae-ad26-6f1e19915ffd" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932893 4860 cpu_manager.go:414] "containerMap: removing container" podUID="9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.932900 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933025 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="76282517-6c00-4e83-b5f5-a6cfaa0dcad2" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933040 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c12481ab-3025-40e3-a9a7-50b44df24d70" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933050 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6975e1bd-ced6-4be9-8808-2e6fa6ff2a89" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933058 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e31cdb01-9f71-4eae-ad26-6f1e19915ffd" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933070 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e2cf636-17c0-4940-84d5-4e7f9faa7dc9" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933079 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="82dde012-8d96-4cc2-8887-99d8e9833f23" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933087 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e2db1f-bd9c-46f0-b913-9533fcf95a4d" containerName="payload-pre-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933096 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b2fd7d0-d1d9-4087-9d88-25142f4a6b65" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933105 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb27095-a4ff-429a-91be-6f3e28872f18" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933113 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d179ee-224e-4375-8cf1-863a774e1cb6" containerName="payload-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933121 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf880079-551f-4973-ab8b-4009619a4fe5" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933130 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e50a1a79-1669-4ad8-89bb-d462a0e80100" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933138 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4332a04-bc02-4fce-925b-df037762c76f" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933145 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a632f9-0197-4f8c-b414-c15b29caea6e" containerName="discovery" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933153 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b585147-38f8-4a8f-be0b-d1ffaef9481d" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933163 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="12ad75be-cbdb-4b90-98d2-7b09cb7cbcc5" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933171 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ef1178-de19-4f53-9646-f4ac3ed52f57" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933181 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c92d9c87-7ba0-4d6e-b659-9e916730c79e" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933189 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0839624-4c7c-42fe-acc1-7515cea20996" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933196 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b05251-7074-41bc-8994-c1051abee475" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933204 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6eb0e0c-ec0c-4cf6-bf46-972c912fd7c4" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933213 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e30f93-2270-4fb7-bfc2-3e48edf5ecc1" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933223 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67e8cb6-152f-4cdd-a814-d1cfdbf99d96" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933232 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7df1cfb-533d-4c2b-ad3c-1dbec34b0388" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933240 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="151de229-e8e3-4bc3-8443-b12cfca9aab8" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933248 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce731d39-0cbe-4322-aa80-1d2606c3970a" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933255 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="66504ddb-d869-4549-b870-61b42e58e871" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933262 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9293d64-b25e-4076-b707-fc65717da6a8" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933273 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="cef174d1-f0e7-4748-997a-fa814937890c" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933284 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc8a063b-3299-4bb8-a4d8-1eb79e6adca3" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933293 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a27864d8-7810-45a3-8cf1-ccc0033b4268" containerName="payload-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933303 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b05251-7074-41bc-8994-c1051abee475" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933311 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="edbe40ed-0184-44e4-990c-2b7a3fd51f5f" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933318 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="78574547-89da-456a-a2ca-137fba06a5b3" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933327 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b53fb7b2-d91c-409b-bf83-e24b95a7608f" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933335 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebd4fcc7-38eb-40ff-b389-8b1fce29d132" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933344 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="63743df5-0f1d-4708-8c41-80b2c9344fc5" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933354 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="39a03fdb-9060-4f2c-a96e-d78764568df8" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933363 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95eaf1d-668d-405f-bbc1-6be67fbe1fed" containerName="payload-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933373 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e58f61-43a2-4600-a380-549625283343" containerName="payload-pre-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933381 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="355a0269-1933-46ee-b701-5a5c45da4061" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933390 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8332d1-5646-4263-957d-58fa676e8775" containerName="istio-proxy" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933399 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="719f739a-813b-4f51-9bd4-e523bba953af" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933407 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6277b932-8a69-4947-afe2-0946a7c6f177" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933415 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="bafb2593-6516-446c-b113-37bff6320729" containerName="payload-processing" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933422 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="aee9ada9-06c5-48ab-b6e3-195c1b445270" containerName="main" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933429 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="42544ef4-8181-4c6e-bbd8-337a63fc0412" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933437 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a9aa9e4-1f8b-4512-ae25-51b759bd1c91" containerName="maas-api" Nov 02 18:15:54.933343 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933447 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="caee9964-de70-4471-9013-90b120d20fa1" containerName="payload-pre-processing" Nov 02 18:15:54.937798 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933456 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="69187cd1-8708-46d3-91d9-62b9a9446454" containerName="istio-proxy" Nov 02 18:15:54.937798 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933464 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e0682c-4519-452a-ab21-9781064a6260" containerName="maas-api" Nov 02 18:15:54.937798 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933471 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="134da1f2-4c24-495f-9b0a-76d21ba04df4" containerName="maas-api" Nov 02 18:15:54.937798 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933478 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b9f322-2bf8-415b-a26a-3723a34c2482" containerName="maas-api" Nov 02 18:15:54.937798 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933487 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceba04e6-9968-4301-aaa3-6372edea5720" containerName="maas-api" Nov 02 18:15:54.937798 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933496 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="63359d54-2671-4cbc-9033-cc1716d65580" containerName="payload-pre-processing" Nov 02 18:15:54.937798 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933505 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d64cc17-abaf-4e5e-bb0d-b2ed1338437c" containerName="main" Nov 02 18:15:54.937798 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933512 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="980e42a5-a602-46ea-842d-6b6914e1b870" containerName="payload-processing" Nov 02 18:15:54.937798 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.933519 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="840ac652-3ceb-43ef-91b3-42c19179fe01" containerName="payload-pre-processing" Nov 02 18:15:54.938649 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.938627 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj" Nov 02 18:15:54.954898 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:54.954877 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj"] Nov 02 18:15:55.031600 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:55.031566 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:15:55.031600 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:55.031576 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 18:15:55.058652 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:55.058622 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgs7h\" (UniqueName: \"kubernetes.io/projected/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-kube-api-access-xgs7h\") pod \"maas-api-revoke-sub-232313e4-95zzj\" (UID: \"08d029a4-bde0-4ff2-8d27-fa04d90dd1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj" Nov 02 18:15:55.058805 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:55.058777 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-232313e4-95zzj\" (UID: \"08d029a4-bde0-4ff2-8d27-fa04d90dd1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj" Nov 02 18:15:55.058879 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:55.058863 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-232313e4-95zzj\" (UID: \"08d029a4-bde0-4ff2-8d27-fa04d90dd1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj" Nov 02 18:15:55.159588 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:55.159552 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-232313e4-95zzj\" (UID: \"08d029a4-bde0-4ff2-8d27-fa04d90dd1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj" Nov 02 18:15:55.159772 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:55.159635 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-232313e4-95zzj\" (UID: \"08d029a4-bde0-4ff2-8d27-fa04d90dd1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj" Nov 02 18:15:55.159772 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:55.159673 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgs7h\" (UniqueName: \"kubernetes.io/projected/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-kube-api-access-xgs7h\") pod \"maas-api-revoke-sub-232313e4-95zzj\" (UID: \"08d029a4-bde0-4ff2-8d27-fa04d90dd1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj" Nov 02 18:15:55.160355 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:55.160328 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-232313e4-95zzj\" (UID: \"08d029a4-bde0-4ff2-8d27-fa04d90dd1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj" Nov 02 18:15:55.162068 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:55.162039 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-232313e4-95zzj\" (UID: \"08d029a4-bde0-4ff2-8d27-fa04d90dd1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj" Nov 02 18:15:55.186190 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:55.186122 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgs7h\" (UniqueName: \"kubernetes.io/projected/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-kube-api-access-xgs7h\") pod \"maas-api-revoke-sub-232313e4-95zzj\" (UID: \"08d029a4-bde0-4ff2-8d27-fa04d90dd1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj" Nov 02 18:15:55.252711 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:55.252679 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj" Nov 02 18:15:55.553813 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:55.553784 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj"] Nov 02 18:15:55.555350 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:15:55.555321 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d029a4_bde0_4ff2_8d27_fa04d90dd1d1.slice/crio-6f309f3c5d85c96fd89121b52e7f993bced2179e80122140bb5c5bf7bc1234df WatchSource:0}: Error finding container 6f309f3c5d85c96fd89121b52e7f993bced2179e80122140bb5c5bf7bc1234df: Status 404 returned error can't find the container with id 6f309f3c5d85c96fd89121b52e7f993bced2179e80122140bb5c5bf7bc1234df Nov 02 18:15:56.319159 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:56.319119 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj" event={"ID":"08d029a4-bde0-4ff2-8d27-fa04d90dd1d1","Type":"ContainerStarted","Data":"06c9172e450733f95dbc9624a7521bfed9758bc70b0d58ba34c2ff6a73263327"} Nov 02 18:15:56.319569 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:56.319181 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj" event={"ID":"08d029a4-bde0-4ff2-8d27-fa04d90dd1d1","Type":"ContainerStarted","Data":"6f309f3c5d85c96fd89121b52e7f993bced2179e80122140bb5c5bf7bc1234df"} Nov 02 18:15:56.359116 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:56.359064 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj" podStartSLOduration=2.359040098 podStartE2EDuration="2.359040098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:56.358944865 +0000 UTC m=+2774.980304642" watchObservedRunningTime="2026-11-02 18:15:56.359040098 +0000 UTC m=+2774.980399871" Nov 02 18:15:57.326247 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:57.326213 4860 generic.go:356] "Generic (PLEG): container finished" podID="08d029a4-bde0-4ff2-8d27-fa04d90dd1d1" containerID="06c9172e450733f95dbc9624a7521bfed9758bc70b0d58ba34c2ff6a73263327" exitCode=0 Nov 02 18:15:57.326706 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:57.326302 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj" event={"ID":"08d029a4-bde0-4ff2-8d27-fa04d90dd1d1","Type":"ContainerDied","Data":"06c9172e450733f95dbc9624a7521bfed9758bc70b0d58ba34c2ff6a73263327"} Nov 02 18:15:57.385652 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:57.385593 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" podUID="b51de62e-89f5-4ac1-9f79-b64defc7e887" containerName="payload-processing" containerID="cri-o://bc3c8d206613a2c3a47429339ebf144a63d7c862f515967e8a9996b6f81fc386" gracePeriod=25 Nov 02 18:15:57.431407 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:57.431346 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" podUID="c40d2ec9-ed70-417a-8c81-2a93ae477425" containerName="payload-pre-processing" containerID="cri-o://422e6754eac6b47f2673e24062270d5b94a01b65f3ba7ebfd1ccdc50c06420c5" gracePeriod=25 Nov 02 18:15:58.468233 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:58.468205 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj" Nov 02 18:15:58.589980 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:58.589890 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-maas-api-openshift-ca\") pod \"08d029a4-bde0-4ff2-8d27-fa04d90dd1d1\" (UID: \"08d029a4-bde0-4ff2-8d27-fa04d90dd1d1\") " Nov 02 18:15:58.589980 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:58.589933 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-maas-api-xks-ca\") pod \"08d029a4-bde0-4ff2-8d27-fa04d90dd1d1\" (UID: \"08d029a4-bde0-4ff2-8d27-fa04d90dd1d1\") " Nov 02 18:15:58.589980 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:58.589970 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-kube-api-access-xgs7h\" (UniqueName: \"kubernetes.io/projected/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-kube-api-access-xgs7h\") pod \"08d029a4-bde0-4ff2-8d27-fa04d90dd1d1\" (UID: \"08d029a4-bde0-4ff2-8d27-fa04d90dd1d1\") " Nov 02 18:15:58.590267 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:58.590237 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-maas-api-openshift-ca" pod "08d029a4-bde0-4ff2-8d27-fa04d90dd1d1" (UID: "08d029a4-bde0-4ff2-8d27-fa04d90dd1d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:58.590474 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:58.590455 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:15:58.592222 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:58.592196 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-maas-api-xks-ca" pod "08d029a4-bde0-4ff2-8d27-fa04d90dd1d1" (UID: "08d029a4-bde0-4ff2-8d27-fa04d90dd1d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:15:58.592222 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:58.592221 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-kube-api-access-xgs7h" pod "08d029a4-bde0-4ff2-8d27-fa04d90dd1d1" (UID: "08d029a4-bde0-4ff2-8d27-fa04d90dd1d1"). InnerVolumeSpecName "kube-api-access-xgs7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:58.691340 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:58.691312 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:15:58.691340 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:58.691338 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgs7h\" (UniqueName: \"kubernetes.io/projected/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1-kube-api-access-xgs7h\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:15:59.340537 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:59.340505 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj" event={"ID":"08d029a4-bde0-4ff2-8d27-fa04d90dd1d1","Type":"ContainerDied","Data":"6f309f3c5d85c96fd89121b52e7f993bced2179e80122140bb5c5bf7bc1234df"} Nov 02 18:15:59.340537 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:59.340538 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj" Nov 02 18:15:59.340763 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:15:59.340552 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f309f3c5d85c96fd89121b52e7f993bced2179e80122140bb5c5bf7bc1234df" Nov 02 18:16:01.322160 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:01.322118 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" podUID="c40d2ec9-ed70-417a-8c81-2a93ae477425" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:16:01.322624 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:01.322206 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" podUID="b51de62e-89f5-4ac1-9f79-b64defc7e887" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:16:01.637904 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:01.637806 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" containerID="cri-o://d94f88a65fd5635737b33220a7c3d145621277bb49f96f0b0d3e50d548dbe73c" gracePeriod=45 Nov 02 18:16:01.652436 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:01.652372 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" containerID="cri-o://b7662f9d1ef269d306e89d58541dbe51a4c94714d5799bad4bbb00233092ac6e" gracePeriod=45 Nov 02 18:16:01.659448 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:01.659421 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" containerID="cri-o://1dfcd2cecffe65cbd9feadf43f217de3c10e8edb461d22eeb99da7f151fe260e" gracePeriod=45 Nov 02 18:16:01.812640 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:01.812592 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": dial tcp 10.132.0.99:8000: connect: connection refused" Nov 02 18:16:01.935360 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:01.935333 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 18:16:01.950877 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:01.950827 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 18:16:01.962142 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:01.962121 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 18:16:02.028126 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.028077 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-tmp-dir\") pod \"fe8fb265-b2fa-44a3-8960-834a02db7553\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " Nov 02 18:16:02.028319 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.028160 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-home\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-home\") pod \"fe8fb265-b2fa-44a3-8960-834a02db7553\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " Nov 02 18:16:02.028319 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.028211 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76b70b32-bbca-4658-97e0-e8c0e7927307-tls-certs\" (UniqueName: \"kubernetes.io/secret/76b70b32-bbca-4658-97e0-e8c0e7927307-tls-certs\") pod \"76b70b32-bbca-4658-97e0-e8c0e7927307\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " Nov 02 18:16:02.028319 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.028263 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe8fb265-b2fa-44a3-8960-834a02db7553-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8fb265-b2fa-44a3-8960-834a02db7553-tls-certs\") pod \"fe8fb265-b2fa-44a3-8960-834a02db7553\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " Nov 02 18:16:02.028513 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.028329 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-home\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-home\") pod \"76b70b32-bbca-4658-97e0-e8c0e7927307\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " Nov 02 18:16:02.028513 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.028359 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8fb265-b2fa-44a3-8960-834a02db7553-kube-api-access-zxdjb\" (UniqueName: \"kubernetes.io/projected/fe8fb265-b2fa-44a3-8960-834a02db7553-kube-api-access-zxdjb\") pod \"fe8fb265-b2fa-44a3-8960-834a02db7553\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " Nov 02 18:16:02.028513 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.028417 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-dshm\") pod \"fe8fb265-b2fa-44a3-8960-834a02db7553\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " Nov 02 18:16:02.029084 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.028742 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-tmp-dir" pod "fe8fb265-b2fa-44a3-8960-834a02db7553" (UID: "fe8fb265-b2fa-44a3-8960-834a02db7553"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:02.029084 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.028758 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-home" pod "fe8fb265-b2fa-44a3-8960-834a02db7553" (UID: "fe8fb265-b2fa-44a3-8960-834a02db7553"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:02.029329 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.029189 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-home" pod "76b70b32-bbca-4658-97e0-e8c0e7927307" (UID: "76b70b32-bbca-4658-97e0-e8c0e7927307"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:02.029329 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.029278 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-model-cache\") pod \"fe8fb265-b2fa-44a3-8960-834a02db7553\" (UID: \"fe8fb265-b2fa-44a3-8960-834a02db7553\") " Nov 02 18:16:02.029446 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.029352 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b70b32-bbca-4658-97e0-e8c0e7927307-kube-api-access-rl9qf\" (UniqueName: \"kubernetes.io/projected/76b70b32-bbca-4658-97e0-e8c0e7927307-kube-api-access-rl9qf\") pod \"76b70b32-bbca-4658-97e0-e8c0e7927307\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " Nov 02 18:16:02.029546 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.029504 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-model-cache" pod "fe8fb265-b2fa-44a3-8960-834a02db7553" (UID: "fe8fb265-b2fa-44a3-8960-834a02db7553"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:02.029763 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.029741 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-dshm\") pod \"76b70b32-bbca-4658-97e0-e8c0e7927307\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " Nov 02 18:16:02.029966 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.029806 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-model-cache\") pod \"76b70b32-bbca-4658-97e0-e8c0e7927307\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " Nov 02 18:16:02.030080 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.030058 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-model-cache" pod "76b70b32-bbca-4658-97e0-e8c0e7927307" (UID: "76b70b32-bbca-4658-97e0-e8c0e7927307"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:02.030152 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.029843 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-tmp-dir\") pod \"76b70b32-bbca-4658-97e0-e8c0e7927307\" (UID: \"76b70b32-bbca-4658-97e0-e8c0e7927307\") " Nov 02 18:16:02.030339 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.030319 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-tmp-dir" pod "76b70b32-bbca-4658-97e0-e8c0e7927307" (UID: "76b70b32-bbca-4658-97e0-e8c0e7927307"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:02.030723 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.030700 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:02.030817 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.030726 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:02.030817 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.030740 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:02.030817 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.030753 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:02.030817 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.030765 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:02.030817 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.030777 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:02.031624 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.031600 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76b70b32-bbca-4658-97e0-e8c0e7927307-tls-certs" pod "76b70b32-bbca-4658-97e0-e8c0e7927307" (UID: "76b70b32-bbca-4658-97e0-e8c0e7927307"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:16:02.031942 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.031910 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-dshm" pod "fe8fb265-b2fa-44a3-8960-834a02db7553" (UID: "fe8fb265-b2fa-44a3-8960-834a02db7553"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:02.032045 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.032001 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8fb265-b2fa-44a3-8960-834a02db7553-kube-api-access-zxdjb" pod "fe8fb265-b2fa-44a3-8960-834a02db7553" (UID: "fe8fb265-b2fa-44a3-8960-834a02db7553"). InnerVolumeSpecName "kube-api-access-zxdjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:02.032763 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.032732 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe8fb265-b2fa-44a3-8960-834a02db7553-tls-certs" pod "fe8fb265-b2fa-44a3-8960-834a02db7553" (UID: "fe8fb265-b2fa-44a3-8960-834a02db7553"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:16:02.033053 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.033028 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b70b32-bbca-4658-97e0-e8c0e7927307-kube-api-access-rl9qf" pod "76b70b32-bbca-4658-97e0-e8c0e7927307" (UID: "76b70b32-bbca-4658-97e0-e8c0e7927307"). InnerVolumeSpecName "kube-api-access-rl9qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:02.033504 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.033482 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-dshm" pod "76b70b32-bbca-4658-97e0-e8c0e7927307" (UID: "76b70b32-bbca-4658-97e0-e8c0e7927307"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:02.131839 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.131801 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-home\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-home\") pod \"b52df912-8bb6-4b46-b2bc-c51da240b140\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " Nov 02 18:16:02.132042 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.131868 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-model-cache\") pod \"b52df912-8bb6-4b46-b2bc-c51da240b140\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " Nov 02 18:16:02.132042 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.131918 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-tmp-dir\") pod \"b52df912-8bb6-4b46-b2bc-c51da240b140\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " Nov 02 18:16:02.132042 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.131964 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-dshm\") pod \"b52df912-8bb6-4b46-b2bc-c51da240b140\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " Nov 02 18:16:02.132225 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.132049 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b52df912-8bb6-4b46-b2bc-c51da240b140-tls-certs\" (UniqueName: \"kubernetes.io/secret/b52df912-8bb6-4b46-b2bc-c51da240b140-tls-certs\") pod \"b52df912-8bb6-4b46-b2bc-c51da240b140\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " Nov 02 18:16:02.132225 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.132080 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b52df912-8bb6-4b46-b2bc-c51da240b140-kube-api-access-w8jlh\" (UniqueName: \"kubernetes.io/projected/b52df912-8bb6-4b46-b2bc-c51da240b140-kube-api-access-w8jlh\") pod \"b52df912-8bb6-4b46-b2bc-c51da240b140\" (UID: \"b52df912-8bb6-4b46-b2bc-c51da240b140\") " Nov 02 18:16:02.132225 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.132175 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-home" pod "b52df912-8bb6-4b46-b2bc-c51da240b140" (UID: "b52df912-8bb6-4b46-b2bc-c51da240b140"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:02.132225 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.132190 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-model-cache" pod "b52df912-8bb6-4b46-b2bc-c51da240b140" (UID: "b52df912-8bb6-4b46-b2bc-c51da240b140"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:02.132433 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.132249 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-tmp-dir" pod "b52df912-8bb6-4b46-b2bc-c51da240b140" (UID: "b52df912-8bb6-4b46-b2bc-c51da240b140"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:02.132433 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.132408 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:02.132433 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.132420 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8fb265-b2fa-44a3-8960-834a02db7553-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:02.132433 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.132430 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:02.132569 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.132439 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl9qf\" (UniqueName: \"kubernetes.io/projected/76b70b32-bbca-4658-97e0-e8c0e7927307-kube-api-access-rl9qf\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:02.132569 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.132449 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b70b32-bbca-4658-97e0-e8c0e7927307-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:02.132569 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.132458 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b70b32-bbca-4658-97e0-e8c0e7927307-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:02.132569 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.132467 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8fb265-b2fa-44a3-8960-834a02db7553-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:02.132569 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.132476 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:02.132569 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.132495 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxdjb\" (UniqueName: \"kubernetes.io/projected/fe8fb265-b2fa-44a3-8960-834a02db7553-kube-api-access-zxdjb\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:02.134163 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.134139 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-dshm" pod "b52df912-8bb6-4b46-b2bc-c51da240b140" (UID: "b52df912-8bb6-4b46-b2bc-c51da240b140"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:02.134240 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.134224 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b52df912-8bb6-4b46-b2bc-c51da240b140-tls-certs" pod "b52df912-8bb6-4b46-b2bc-c51da240b140" (UID: "b52df912-8bb6-4b46-b2bc-c51da240b140"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:16:02.134281 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.134232 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b52df912-8bb6-4b46-b2bc-c51da240b140-kube-api-access-w8jlh" pod "b52df912-8bb6-4b46-b2bc-c51da240b140" (UID: "b52df912-8bb6-4b46-b2bc-c51da240b140"). InnerVolumeSpecName "kube-api-access-w8jlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:02.233258 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.233165 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b52df912-8bb6-4b46-b2bc-c51da240b140-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:02.233258 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.233202 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8jlh\" (UniqueName: \"kubernetes.io/projected/b52df912-8bb6-4b46-b2bc-c51da240b140-kube-api-access-w8jlh\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:02.233258 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.233216 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b52df912-8bb6-4b46-b2bc-c51da240b140-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:02.363692 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.363649 4860 generic.go:356] "Generic (PLEG): container finished" podID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerID="b7662f9d1ef269d306e89d58541dbe51a4c94714d5799bad4bbb00233092ac6e" exitCode=0 Nov 02 18:16:02.364163 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.363699 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" Nov 02 18:16:02.364163 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.363710 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" event={"ID":"76b70b32-bbca-4658-97e0-e8c0e7927307","Type":"ContainerDied","Data":"b7662f9d1ef269d306e89d58541dbe51a4c94714d5799bad4bbb00233092ac6e"} Nov 02 18:16:02.364163 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.363759 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65" event={"ID":"76b70b32-bbca-4658-97e0-e8c0e7927307","Type":"ContainerDied","Data":"6aabc715ffb6732b81670b2d57fc72a617013c665033c55e74e757c7bf595a2f"} Nov 02 18:16:02.364163 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.363782 4860 scope.go:122] "RemoveContainer" containerID="b7662f9d1ef269d306e89d58541dbe51a4c94714d5799bad4bbb00233092ac6e" Nov 02 18:16:02.365271 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.365248 4860 generic.go:356] "Generic (PLEG): container finished" podID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerID="1dfcd2cecffe65cbd9feadf43f217de3c10e8edb461d22eeb99da7f151fe260e" exitCode=0 Nov 02 18:16:02.365403 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.365368 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" event={"ID":"b52df912-8bb6-4b46-b2bc-c51da240b140","Type":"ContainerDied","Data":"1dfcd2cecffe65cbd9feadf43f217de3c10e8edb461d22eeb99da7f151fe260e"} Nov 02 18:16:02.365403 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.365392 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" event={"ID":"b52df912-8bb6-4b46-b2bc-c51da240b140","Type":"ContainerDied","Data":"9827f5a6b4dd6c73db8306d6af8ce58916ca6b773870d0e4c6a71694fd8ff124"} Nov 02 18:16:02.365529 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.365438 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw" Nov 02 18:16:02.369396 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.369368 4860 generic.go:356] "Generic (PLEG): container finished" podID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerID="d94f88a65fd5635737b33220a7c3d145621277bb49f96f0b0d3e50d548dbe73c" exitCode=0 Nov 02 18:16:02.369524 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.369405 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" Nov 02 18:16:02.369524 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.369448 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" event={"ID":"fe8fb265-b2fa-44a3-8960-834a02db7553","Type":"ContainerDied","Data":"d94f88a65fd5635737b33220a7c3d145621277bb49f96f0b0d3e50d548dbe73c"} Nov 02 18:16:02.369524 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.369489 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc" event={"ID":"fe8fb265-b2fa-44a3-8960-834a02db7553","Type":"ContainerDied","Data":"40166d0742079a1e4235fd84ba14bf8b9e8384137cbe691833f4a576728d974f"} Nov 02 18:16:02.374900 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.374879 4860 scope.go:122] "RemoveContainer" containerID="b7662f9d1ef269d306e89d58541dbe51a4c94714d5799bad4bbb00233092ac6e" Nov 02 18:16:02.375186 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:16:02.375165 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7662f9d1ef269d306e89d58541dbe51a4c94714d5799bad4bbb00233092ac6e\": container with ID starting with b7662f9d1ef269d306e89d58541dbe51a4c94714d5799bad4bbb00233092ac6e not found: ID does not exist" containerID="b7662f9d1ef269d306e89d58541dbe51a4c94714d5799bad4bbb00233092ac6e" Nov 02 18:16:02.375283 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.375194 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7662f9d1ef269d306e89d58541dbe51a4c94714d5799bad4bbb00233092ac6e"} err="failed to get container status \"b7662f9d1ef269d306e89d58541dbe51a4c94714d5799bad4bbb00233092ac6e\": rpc error: code = NotFound desc = could not find container \"b7662f9d1ef269d306e89d58541dbe51a4c94714d5799bad4bbb00233092ac6e\": container with ID starting with b7662f9d1ef269d306e89d58541dbe51a4c94714d5799bad4bbb00233092ac6e not found: ID does not exist" Nov 02 18:16:02.375283 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.375213 4860 scope.go:122] "RemoveContainer" containerID="1dfcd2cecffe65cbd9feadf43f217de3c10e8edb461d22eeb99da7f151fe260e" Nov 02 18:16:02.384210 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.384192 4860 scope.go:122] "RemoveContainer" containerID="1dfcd2cecffe65cbd9feadf43f217de3c10e8edb461d22eeb99da7f151fe260e" Nov 02 18:16:02.384447 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:16:02.384429 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dfcd2cecffe65cbd9feadf43f217de3c10e8edb461d22eeb99da7f151fe260e\": container with ID starting with 1dfcd2cecffe65cbd9feadf43f217de3c10e8edb461d22eeb99da7f151fe260e not found: ID does not exist" containerID="1dfcd2cecffe65cbd9feadf43f217de3c10e8edb461d22eeb99da7f151fe260e" Nov 02 18:16:02.384489 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.384456 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfcd2cecffe65cbd9feadf43f217de3c10e8edb461d22eeb99da7f151fe260e"} err="failed to get container status \"1dfcd2cecffe65cbd9feadf43f217de3c10e8edb461d22eeb99da7f151fe260e\": rpc error: code = NotFound desc = could not find container \"1dfcd2cecffe65cbd9feadf43f217de3c10e8edb461d22eeb99da7f151fe260e\": container with ID starting with 1dfcd2cecffe65cbd9feadf43f217de3c10e8edb461d22eeb99da7f151fe260e not found: ID does not exist" Nov 02 18:16:02.384489 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.384473 4860 scope.go:122] "RemoveContainer" containerID="d94f88a65fd5635737b33220a7c3d145621277bb49f96f0b0d3e50d548dbe73c" Nov 02 18:16:02.392798 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.392780 4860 scope.go:122] "RemoveContainer" containerID="d94f88a65fd5635737b33220a7c3d145621277bb49f96f0b0d3e50d548dbe73c" Nov 02 18:16:02.393099 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:16:02.393081 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d94f88a65fd5635737b33220a7c3d145621277bb49f96f0b0d3e50d548dbe73c\": container with ID starting with d94f88a65fd5635737b33220a7c3d145621277bb49f96f0b0d3e50d548dbe73c not found: ID does not exist" containerID="d94f88a65fd5635737b33220a7c3d145621277bb49f96f0b0d3e50d548dbe73c" Nov 02 18:16:02.393150 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.393105 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94f88a65fd5635737b33220a7c3d145621277bb49f96f0b0d3e50d548dbe73c"} err="failed to get container status \"d94f88a65fd5635737b33220a7c3d145621277bb49f96f0b0d3e50d548dbe73c\": rpc error: code = NotFound desc = could not find container \"d94f88a65fd5635737b33220a7c3d145621277bb49f96f0b0d3e50d548dbe73c\": container with ID starting with d94f88a65fd5635737b33220a7c3d145621277bb49f96f0b0d3e50d548dbe73c not found: ID does not exist" Nov 02 18:16:02.417091 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.417061 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw"] Nov 02 18:16:02.420776 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.420752 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/simulator-w0-069fa5-kserve-79d59c7ff8-dt6cw"] Nov 02 18:16:02.455357 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.455318 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc"] Nov 02 18:16:02.459026 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.459001 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/distinct-2-w0-069fa5-kserve-86775c78fc-nhxwc"] Nov 02 18:16:02.493204 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.493114 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65"] Nov 02 18:16:02.498789 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:02.498764 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/distinct-w0-069fa5-kserve-5d5d4cc7b6-25h65"] Nov 02 18:16:03.497409 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:03.497378 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" path="/var/lib/kubelet/pods/76b70b32-bbca-4658-97e0-e8c0e7927307/volumes" Nov 02 18:16:03.498023 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:03.498005 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" path="/var/lib/kubelet/pods/b52df912-8bb6-4b46-b2bc-c51da240b140/volumes" Nov 02 18:16:03.498451 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:03.498437 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" path="/var/lib/kubelet/pods/fe8fb265-b2fa-44a3-8960-834a02db7553/volumes" Nov 02 18:16:11.321077 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:11.321034 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" podUID="c40d2ec9-ed70-417a-8c81-2a93ae477425" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:16:11.321561 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:11.321119 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" podUID="b51de62e-89f5-4ac1-9f79-b64defc7e887" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:16:17.702110 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:17.702079 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" Nov 02 18:16:17.718626 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:17.718593 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" Nov 02 18:16:17.778431 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:17.778401 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40d2ec9-ed70-417a-8c81-2a93ae477425-kube-api-access-jffb6\" (UniqueName: \"kubernetes.io/projected/c40d2ec9-ed70-417a-8c81-2a93ae477425-kube-api-access-jffb6\") pod \"c40d2ec9-ed70-417a-8c81-2a93ae477425\" (UID: \"c40d2ec9-ed70-417a-8c81-2a93ae477425\") " Nov 02 18:16:17.778611 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:17.778439 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40d2ec9-ed70-417a-8c81-2a93ae477425-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c40d2ec9-ed70-417a-8c81-2a93ae477425-plugins-config-volume\") pod \"c40d2ec9-ed70-417a-8c81-2a93ae477425\" (UID: \"c40d2ec9-ed70-417a-8c81-2a93ae477425\") " Nov 02 18:16:17.778611 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:17.778494 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b51de62e-89f5-4ac1-9f79-b64defc7e887-kube-api-access-brnvz\" (UniqueName: \"kubernetes.io/projected/b51de62e-89f5-4ac1-9f79-b64defc7e887-kube-api-access-brnvz\") pod \"b51de62e-89f5-4ac1-9f79-b64defc7e887\" (UID: \"b51de62e-89f5-4ac1-9f79-b64defc7e887\") " Nov 02 18:16:17.778611 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:17.778560 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b51de62e-89f5-4ac1-9f79-b64defc7e887-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51de62e-89f5-4ac1-9f79-b64defc7e887-plugins-config-volume\") pod \"b51de62e-89f5-4ac1-9f79-b64defc7e887\" (UID: \"b51de62e-89f5-4ac1-9f79-b64defc7e887\") " Nov 02 18:16:17.778996 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:17.778961 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40d2ec9-ed70-417a-8c81-2a93ae477425-plugins-config-volume" pod "c40d2ec9-ed70-417a-8c81-2a93ae477425" (UID: "c40d2ec9-ed70-417a-8c81-2a93ae477425"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:16:17.779159 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:17.779059 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b51de62e-89f5-4ac1-9f79-b64defc7e887-plugins-config-volume" pod "b51de62e-89f5-4ac1-9f79-b64defc7e887" (UID: "b51de62e-89f5-4ac1-9f79-b64defc7e887"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:16:17.780722 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:17.780694 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40d2ec9-ed70-417a-8c81-2a93ae477425-kube-api-access-jffb6" pod "c40d2ec9-ed70-417a-8c81-2a93ae477425" (UID: "c40d2ec9-ed70-417a-8c81-2a93ae477425"). InnerVolumeSpecName "kube-api-access-jffb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:17.780841 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:17.780795 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51de62e-89f5-4ac1-9f79-b64defc7e887-kube-api-access-brnvz" pod "b51de62e-89f5-4ac1-9f79-b64defc7e887" (UID: "b51de62e-89f5-4ac1-9f79-b64defc7e887"). InnerVolumeSpecName "kube-api-access-brnvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:17.880562 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:17.880519 4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c40d2ec9-ed70-417a-8c81-2a93ae477425-plugins-config-volume\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:17.880562 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:17.880562 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brnvz\" (UniqueName: \"kubernetes.io/projected/b51de62e-89f5-4ac1-9f79-b64defc7e887-kube-api-access-brnvz\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:17.880776 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:17.880577 4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51de62e-89f5-4ac1-9f79-b64defc7e887-plugins-config-volume\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:17.880776 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:17.880591 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jffb6\" (UniqueName: \"kubernetes.io/projected/c40d2ec9-ed70-417a-8c81-2a93ae477425-kube-api-access-jffb6\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:18.154603 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.154562 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462"] Nov 02 18:16:18.154876 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.154824 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" podUID="fa5e98eb-b605-4c58-aec6-e10ac2a890d6" containerName="istio-proxy" containerID="cri-o://9fcf80adab1412996324f13f30f98a389492b9306cbb741ffbdcfdc23de17b41" gracePeriod=30 Nov 02 18:16:18.478107 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.478020 4860 generic.go:356] "Generic (PLEG): container finished" podID="b51de62e-89f5-4ac1-9f79-b64defc7e887" containerID="bc3c8d206613a2c3a47429339ebf144a63d7c862f515967e8a9996b6f81fc386" exitCode=0 Nov 02 18:16:18.478107 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.478067 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" Nov 02 18:16:18.478107 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.478089 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" event={"ID":"b51de62e-89f5-4ac1-9f79-b64defc7e887","Type":"ContainerDied","Data":"bc3c8d206613a2c3a47429339ebf144a63d7c862f515967e8a9996b6f81fc386"} Nov 02 18:16:18.478419 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.478145 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp" event={"ID":"b51de62e-89f5-4ac1-9f79-b64defc7e887","Type":"ContainerDied","Data":"ef533dcbe4a136181501e3bab404e5c442130923dcccf0da172724aedd88d92e"} Nov 02 18:16:18.478419 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.478171 4860 scope.go:122] "RemoveContainer" containerID="bc3c8d206613a2c3a47429339ebf144a63d7c862f515967e8a9996b6f81fc386" Nov 02 18:16:18.479515 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.479490 4860 generic.go:356] "Generic (PLEG): container finished" podID="c40d2ec9-ed70-417a-8c81-2a93ae477425" containerID="422e6754eac6b47f2673e24062270d5b94a01b65f3ba7ebfd1ccdc50c06420c5" exitCode=0 Nov 02 18:16:18.479667 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.479532 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" event={"ID":"c40d2ec9-ed70-417a-8c81-2a93ae477425","Type":"ContainerDied","Data":"422e6754eac6b47f2673e24062270d5b94a01b65f3ba7ebfd1ccdc50c06420c5"} Nov 02 18:16:18.479667 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.479540 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" Nov 02 18:16:18.479667 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.479565 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2" event={"ID":"c40d2ec9-ed70-417a-8c81-2a93ae477425","Type":"ContainerDied","Data":"38f9f7dfd2e32792d5d51863d14b0ce6d6eb61d6e1c98a6b4276ce25814b3a70"} Nov 02 18:16:18.489795 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.489761 4860 scope.go:122] "RemoveContainer" containerID="bc3c8d206613a2c3a47429339ebf144a63d7c862f515967e8a9996b6f81fc386" Nov 02 18:16:18.490108 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:16:18.490043 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc3c8d206613a2c3a47429339ebf144a63d7c862f515967e8a9996b6f81fc386\": container with ID starting with bc3c8d206613a2c3a47429339ebf144a63d7c862f515967e8a9996b6f81fc386 not found: ID does not exist" containerID="bc3c8d206613a2c3a47429339ebf144a63d7c862f515967e8a9996b6f81fc386" Nov 02 18:16:18.490108 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.490073 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3c8d206613a2c3a47429339ebf144a63d7c862f515967e8a9996b6f81fc386"} err="failed to get container status \"bc3c8d206613a2c3a47429339ebf144a63d7c862f515967e8a9996b6f81fc386\": rpc error: code = NotFound desc = could not find container \"bc3c8d206613a2c3a47429339ebf144a63d7c862f515967e8a9996b6f81fc386\": container with ID starting with bc3c8d206613a2c3a47429339ebf144a63d7c862f515967e8a9996b6f81fc386 not found: ID does not exist" Nov 02 18:16:18.490108 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.490089 4860 scope.go:122] "RemoveContainer" containerID="422e6754eac6b47f2673e24062270d5b94a01b65f3ba7ebfd1ccdc50c06420c5" Nov 02 18:16:18.499476 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.499459 4860 scope.go:122] "RemoveContainer" containerID="422e6754eac6b47f2673e24062270d5b94a01b65f3ba7ebfd1ccdc50c06420c5" Nov 02 18:16:18.499731 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:16:18.499716 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"422e6754eac6b47f2673e24062270d5b94a01b65f3ba7ebfd1ccdc50c06420c5\": container with ID starting with 422e6754eac6b47f2673e24062270d5b94a01b65f3ba7ebfd1ccdc50c06420c5 not found: ID does not exist" containerID="422e6754eac6b47f2673e24062270d5b94a01b65f3ba7ebfd1ccdc50c06420c5" Nov 02 18:16:18.499779 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.499737 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"422e6754eac6b47f2673e24062270d5b94a01b65f3ba7ebfd1ccdc50c06420c5"} err="failed to get container status \"422e6754eac6b47f2673e24062270d5b94a01b65f3ba7ebfd1ccdc50c06420c5\": rpc error: code = NotFound desc = could not find container \"422e6754eac6b47f2673e24062270d5b94a01b65f3ba7ebfd1ccdc50c06420c5\": container with ID starting with 422e6754eac6b47f2673e24062270d5b94a01b65f3ba7ebfd1ccdc50c06420c5 not found: ID does not exist" Nov 02 18:16:18.531109 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.531071 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp"] Nov 02 18:16:18.539322 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.539291 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-069fa5-f846dc8b9-2mlgp"] Nov 02 18:16:18.614214 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.614178 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2"] Nov 02 18:16:18.636019 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:18.635982 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-069fa5-f6f864fd7-fx8n2"] Nov 02 18:16:19.497484 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:19.497452 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b51de62e-89f5-4ac1-9f79-b64defc7e887" path="/var/lib/kubelet/pods/b51de62e-89f5-4ac1-9f79-b64defc7e887/volumes" Nov 02 18:16:19.498037 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:19.498006 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c40d2ec9-ed70-417a-8c81-2a93ae477425" path="/var/lib/kubelet/pods/c40d2ec9-ed70-417a-8c81-2a93ae477425/volumes" Nov 02 18:16:21.769021 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:16:21.768985 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:16:23.309612 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.309578 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67b9f44df6-zzhv5"] Nov 02 18:16:23.310099 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.309775 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-67b9f44df6-zzhv5" podUID="e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca" containerName="manager" containerID="cri-o://b260e74f62f4b3be15b494726b015b2a7fce41d3d3f9acdf60d95ba321a53018" gracePeriod=30 Nov 02 18:16:23.446368 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.446331 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 18:16:23.523332 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.523301 4860 generic.go:356] "Generic (PLEG): container finished" podID="fa5e98eb-b605-4c58-aec6-e10ac2a890d6" containerID="9fcf80adab1412996324f13f30f98a389492b9306cbb741ffbdcfdc23de17b41" exitCode=0 Nov 02 18:16:23.523576 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.523366 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" Nov 02 18:16:23.523775 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.523373 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" event={"ID":"fa5e98eb-b605-4c58-aec6-e10ac2a890d6","Type":"ContainerDied","Data":"9fcf80adab1412996324f13f30f98a389492b9306cbb741ffbdcfdc23de17b41"} Nov 02 18:16:23.523888 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.523806 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462" event={"ID":"fa5e98eb-b605-4c58-aec6-e10ac2a890d6","Type":"ContainerDied","Data":"9245de7f505c106738554807ec295ca0816b7e1448ae741f52634e7fb3a4e183"} Nov 02 18:16:23.523888 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.523842 4860 scope.go:122] "RemoveContainer" containerID="9fcf80adab1412996324f13f30f98a389492b9306cbb741ffbdcfdc23de17b41" Nov 02 18:16:23.526328 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.526307 4860 generic.go:356] "Generic (PLEG): container finished" podID="e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca" containerID="b260e74f62f4b3be15b494726b015b2a7fce41d3d3f9acdf60d95ba321a53018" exitCode=0 Nov 02 18:16:23.526481 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.526383 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b9f44df6-zzhv5" event={"ID":"e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca","Type":"ContainerDied","Data":"b260e74f62f4b3be15b494726b015b2a7fce41d3d3f9acdf60d95ba321a53018"} Nov 02 18:16:23.542478 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.542455 4860 scope.go:122] "RemoveContainer" containerID="9fcf80adab1412996324f13f30f98a389492b9306cbb741ffbdcfdc23de17b41" Nov 02 18:16:23.542681 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.542646 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-token\" (UniqueName: \"kubernetes.io/projected/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-token\") pod \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " Nov 02 18:16:23.542778 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.542720 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-envoy\") pod \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " Nov 02 18:16:23.542778 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.542755 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-workload-socket\") pod \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " Nov 02 18:16:23.543224 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.542786 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-workload-certs\") pod \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " Nov 02 18:16:23.543224 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.542819 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-credential-socket\") pod \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " Nov 02 18:16:23.543224 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.542895 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-data\") pod \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " Nov 02 18:16:23.543224 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.542954 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-podinfo\") pod \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " Nov 02 18:16:23.543224 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.542987 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-kube-api-access-fxztg\" (UniqueName: \"kubernetes.io/projected/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-kube-api-access-fxztg\") pod \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " Nov 02 18:16:23.543224 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.543035 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istiod-ca-cert\") pod \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\" (UID: \"fa5e98eb-b605-4c58-aec6-e10ac2a890d6\") " Nov 02 18:16:23.544080 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:16:23.543708 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fcf80adab1412996324f13f30f98a389492b9306cbb741ffbdcfdc23de17b41\": container with ID starting with 9fcf80adab1412996324f13f30f98a389492b9306cbb741ffbdcfdc23de17b41 not found: ID does not exist" containerID="9fcf80adab1412996324f13f30f98a389492b9306cbb741ffbdcfdc23de17b41" Nov 02 18:16:23.544080 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.543754 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fcf80adab1412996324f13f30f98a389492b9306cbb741ffbdcfdc23de17b41"} err="failed to get container status \"9fcf80adab1412996324f13f30f98a389492b9306cbb741ffbdcfdc23de17b41\": rpc error: code = NotFound desc = could not find container \"9fcf80adab1412996324f13f30f98a389492b9306cbb741ffbdcfdc23de17b41\": container with ID starting with 9fcf80adab1412996324f13f30f98a389492b9306cbb741ffbdcfdc23de17b41 not found: ID does not exist" Nov 02 18:16:23.544261 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.544226 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-workload-socket" pod "fa5e98eb-b605-4c58-aec6-e10ac2a890d6" (UID: "fa5e98eb-b605-4c58-aec6-e10ac2a890d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:23.544261 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.544241 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-data" pod "fa5e98eb-b605-4c58-aec6-e10ac2a890d6" (UID: "fa5e98eb-b605-4c58-aec6-e10ac2a890d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:23.544494 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.544457 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-credential-socket" pod "fa5e98eb-b605-4c58-aec6-e10ac2a890d6" (UID: "fa5e98eb-b605-4c58-aec6-e10ac2a890d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:23.544602 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.544504 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-workload-certs" pod "fa5e98eb-b605-4c58-aec6-e10ac2a890d6" (UID: "fa5e98eb-b605-4c58-aec6-e10ac2a890d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:23.544828 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.544810 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istiod-ca-cert" pod "fa5e98eb-b605-4c58-aec6-e10ac2a890d6" (UID: "fa5e98eb-b605-4c58-aec6-e10ac2a890d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:16:23.548827 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.548806 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-kube-api-access-fxztg" pod "fa5e98eb-b605-4c58-aec6-e10ac2a890d6" (UID: "fa5e98eb-b605-4c58-aec6-e10ac2a890d6"). InnerVolumeSpecName "kube-api-access-fxztg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:23.548968 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.548928 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-token" pod "fa5e98eb-b605-4c58-aec6-e10ac2a890d6" (UID: "fa5e98eb-b605-4c58-aec6-e10ac2a890d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:23.548968 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.548935 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-envoy" pod "fa5e98eb-b605-4c58-aec6-e10ac2a890d6" (UID: "fa5e98eb-b605-4c58-aec6-e10ac2a890d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:16:23.550475 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.550445 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-podinfo" pod "fa5e98eb-b605-4c58-aec6-e10ac2a890d6" (UID: "fa5e98eb-b605-4c58-aec6-e10ac2a890d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:16:23.590078 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.590049 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b9f44df6-zzhv5" Nov 02 18:16:23.644701 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.644536 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-metrics-certs\") pod \"e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca\" (UID: \"e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca\") " Nov 02 18:16:23.644701 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.644592 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-webhook-cert\") pod \"e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca\" (UID: \"e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca\") " Nov 02 18:16:23.646533 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.645466 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-kube-api-access-9t229\" (UniqueName: \"kubernetes.io/projected/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-kube-api-access-9t229\") pod \"e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca\" (UID: \"e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca\") " Nov 02 18:16:23.646533 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.645830 4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istiod-ca-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:23.646533 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.645863 4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-token\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:23.646533 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.645877 4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-envoy\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:23.646533 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.645889 4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-workload-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:23.646533 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.645903 4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-workload-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:23.646533 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.645916 4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-credential-socket\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:23.646533 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.645929 4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-data\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:23.646533 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.645942 4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-istio-podinfo\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:23.646533 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.645955 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxztg\" (UniqueName: \"kubernetes.io/projected/fa5e98eb-b605-4c58-aec6-e10ac2a890d6-kube-api-access-fxztg\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:23.679513 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.679421 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-metrics-certs" pod "e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca" (UID: "e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:16:23.681086 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.681031 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-kube-api-access-9t229" pod "e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca" (UID: "e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca"). InnerVolumeSpecName "kube-api-access-9t229". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:23.681329 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.681294 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-webhook-cert" pod "e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca" (UID: "e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:16:23.747455 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.747419 4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-metrics-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:23.747455 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.747450 4860 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-webhook-cert\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:23.747455 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.747460 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t229\" (UniqueName: \"kubernetes.io/projected/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca-kube-api-access-9t229\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:23.871655 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.871620 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462"] Nov 02 18:16:23.879653 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:23.879612 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-s7462"] Nov 02 18:16:24.534742 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:24.534660 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67b9f44df6-zzhv5" Nov 02 18:16:24.534742 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:24.534697 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67b9f44df6-zzhv5" event={"ID":"e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca","Type":"ContainerDied","Data":"075be13926602f638ef5a966b9b995278e87c3ae4954386426835d1b186367f9"} Nov 02 18:16:24.535300 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:24.534763 4860 scope.go:122] "RemoveContainer" containerID="b260e74f62f4b3be15b494726b015b2a7fce41d3d3f9acdf60d95ba321a53018" Nov 02 18:16:24.597612 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:24.597573 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67b9f44df6-zzhv5"] Nov 02 18:16:24.605947 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:24.603900 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-67b9f44df6-zzhv5"] Nov 02 18:16:25.500240 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:25.500199 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca" path="/var/lib/kubelet/pods/e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca/volumes" Nov 02 18:16:25.500840 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:25.500820 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa5e98eb-b605-4c58-aec6-e10ac2a890d6" path="/var/lib/kubelet/pods/fa5e98eb-b605-4c58-aec6-e10ac2a890d6/volumes" Nov 02 18:16:47.017740 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.017706 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm"] Nov 02 18:16:47.018157 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018119 4860 cpu_manager.go:414] "containerMap: removing container" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" Nov 02 18:16:47.018157 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018132 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" Nov 02 18:16:47.018157 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018141 4860 cpu_manager.go:414] "containerMap: removing container" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" Nov 02 18:16:47.018157 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018148 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" Nov 02 18:16:47.018292 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018164 4860 cpu_manager.go:414] "containerMap: removing container" podUID="b51de62e-89f5-4ac1-9f79-b64defc7e887" containerName="payload-processing" Nov 02 18:16:47.018292 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018169 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b51de62e-89f5-4ac1-9f79-b64defc7e887" containerName="payload-processing" Nov 02 18:16:47.018292 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018177 4860 cpu_manager.go:414] "containerMap: removing container" podUID="c40d2ec9-ed70-417a-8c81-2a93ae477425" containerName="payload-pre-processing" Nov 02 18:16:47.018292 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018183 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40d2ec9-ed70-417a-8c81-2a93ae477425" containerName="payload-pre-processing" Nov 02 18:16:47.018292 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018197 4860 cpu_manager.go:414] "containerMap: removing container" podUID="08d029a4-bde0-4ff2-8d27-fa04d90dd1d1" containerName="revoke-keys" Nov 02 18:16:47.018292 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018201 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08d029a4-bde0-4ff2-8d27-fa04d90dd1d1" containerName="revoke-keys" Nov 02 18:16:47.018292 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018208 4860 cpu_manager.go:414] "containerMap: removing container" podUID="fa5e98eb-b605-4c58-aec6-e10ac2a890d6" containerName="istio-proxy" Nov 02 18:16:47.018292 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018213 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa5e98eb-b605-4c58-aec6-e10ac2a890d6" containerName="istio-proxy" Nov 02 18:16:47.018292 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018224 4860 cpu_manager.go:414] "containerMap: removing container" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" Nov 02 18:16:47.018292 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018228 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" Nov 02 18:16:47.018292 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018240 4860 cpu_manager.go:414] "containerMap: removing container" podUID="e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca" containerName="manager" Nov 02 18:16:47.018292 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018245 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca" containerName="manager" Nov 02 18:16:47.018642 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018302 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e55d5ea9-6fb8-4f82-a9a9-076d69a0f0ca" containerName="manager" Nov 02 18:16:47.018642 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018312 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b52df912-8bb6-4b46-b2bc-c51da240b140" containerName="main" Nov 02 18:16:47.018642 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018317 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe8fb265-b2fa-44a3-8960-834a02db7553" containerName="main" Nov 02 18:16:47.018642 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018324 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b51de62e-89f5-4ac1-9f79-b64defc7e887" containerName="payload-processing" Nov 02 18:16:47.018642 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018331 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40d2ec9-ed70-417a-8c81-2a93ae477425" containerName="payload-pre-processing" Nov 02 18:16:47.018642 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018337 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="08d029a4-bde0-4ff2-8d27-fa04d90dd1d1" containerName="revoke-keys" Nov 02 18:16:47.018642 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018342 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa5e98eb-b605-4c58-aec6-e10ac2a890d6" containerName="istio-proxy" Nov 02 18:16:47.018642 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.018349 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="76b70b32-bbca-4658-97e0-e8c0e7927307" containerName="main" Nov 02 18:16:47.023002 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.022985 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm" Nov 02 18:16:47.043600 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.043573 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm"] Nov 02 18:16:47.052873 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.052836 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:16:47.052976 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.052940 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 18:16:47.072042 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.072014 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7e6a04-5ad9-4da7-aee3-571710a3213d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rhmhm\" (UID: \"3d7e6a04-5ad9-4da7-aee3-571710a3213d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm" Nov 02 18:16:47.072160 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.072058 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7e6a04-5ad9-4da7-aee3-571710a3213d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rhmhm\" (UID: \"3d7e6a04-5ad9-4da7-aee3-571710a3213d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm" Nov 02 18:16:47.072160 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.072077 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j7sg\" (UniqueName: \"kubernetes.io/projected/3d7e6a04-5ad9-4da7-aee3-571710a3213d-kube-api-access-5j7sg\") pod \"maas-api-revoke-sub-85237e45-rhmhm\" (UID: \"3d7e6a04-5ad9-4da7-aee3-571710a3213d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm" Nov 02 18:16:47.172641 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.172604 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7e6a04-5ad9-4da7-aee3-571710a3213d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rhmhm\" (UID: \"3d7e6a04-5ad9-4da7-aee3-571710a3213d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm" Nov 02 18:16:47.172826 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.172658 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7e6a04-5ad9-4da7-aee3-571710a3213d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rhmhm\" (UID: \"3d7e6a04-5ad9-4da7-aee3-571710a3213d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm" Nov 02 18:16:47.172826 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.172679 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j7sg\" (UniqueName: \"kubernetes.io/projected/3d7e6a04-5ad9-4da7-aee3-571710a3213d-kube-api-access-5j7sg\") pod \"maas-api-revoke-sub-85237e45-rhmhm\" (UID: \"3d7e6a04-5ad9-4da7-aee3-571710a3213d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm" Nov 02 18:16:47.173310 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.173279 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7e6a04-5ad9-4da7-aee3-571710a3213d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rhmhm\" (UID: \"3d7e6a04-5ad9-4da7-aee3-571710a3213d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm" Nov 02 18:16:47.175160 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.175132 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7e6a04-5ad9-4da7-aee3-571710a3213d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rhmhm\" (UID: \"3d7e6a04-5ad9-4da7-aee3-571710a3213d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm" Nov 02 18:16:47.196293 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.196270 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j7sg\" (UniqueName: \"kubernetes.io/projected/3d7e6a04-5ad9-4da7-aee3-571710a3213d-kube-api-access-5j7sg\") pod \"maas-api-revoke-sub-85237e45-rhmhm\" (UID: \"3d7e6a04-5ad9-4da7-aee3-571710a3213d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm" Nov 02 18:16:47.335633 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.335595 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm" Nov 02 18:16:47.639362 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.639320 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm"] Nov 02 18:16:47.640267 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:16:47.640239 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7e6a04_5ad9_4da7_aee3_571710a3213d.slice/crio-383a91cbbf0df27899630c634470bfd84e9f7449167ffb4bbf8edbcaf6e43c4e WatchSource:0}: Error finding container 383a91cbbf0df27899630c634470bfd84e9f7449167ffb4bbf8edbcaf6e43c4e: Status 404 returned error can't find the container with id 383a91cbbf0df27899630c634470bfd84e9f7449167ffb4bbf8edbcaf6e43c4e Nov 02 18:16:47.678883 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:47.678843 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm" event={"ID":"3d7e6a04-5ad9-4da7-aee3-571710a3213d","Type":"ContainerStarted","Data":"383a91cbbf0df27899630c634470bfd84e9f7449167ffb4bbf8edbcaf6e43c4e"} Nov 02 18:16:48.686659 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:48.686618 4860 generic.go:356] "Generic (PLEG): container finished" podID="3d7e6a04-5ad9-4da7-aee3-571710a3213d" containerID="79a37f25a1c526bf14c3a231eb4b26b2753b045b86edb7f557008e97dc9b11ff" exitCode=0 Nov 02 18:16:48.687121 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:48.686699 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm" event={"ID":"3d7e6a04-5ad9-4da7-aee3-571710a3213d","Type":"ContainerDied","Data":"79a37f25a1c526bf14c3a231eb4b26b2753b045b86edb7f557008e97dc9b11ff"} Nov 02 18:16:49.839549 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:49.839512 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm" Nov 02 18:16:49.897101 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:49.897070 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d7e6a04-5ad9-4da7-aee3-571710a3213d-kube-api-access-5j7sg\" (UniqueName: \"kubernetes.io/projected/3d7e6a04-5ad9-4da7-aee3-571710a3213d-kube-api-access-5j7sg\") pod \"3d7e6a04-5ad9-4da7-aee3-571710a3213d\" (UID: \"3d7e6a04-5ad9-4da7-aee3-571710a3213d\") " Nov 02 18:16:49.897288 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:49.897160 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d7e6a04-5ad9-4da7-aee3-571710a3213d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7e6a04-5ad9-4da7-aee3-571710a3213d-maas-api-openshift-ca\") pod \"3d7e6a04-5ad9-4da7-aee3-571710a3213d\" (UID: \"3d7e6a04-5ad9-4da7-aee3-571710a3213d\") " Nov 02 18:16:49.897288 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:49.897214 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d7e6a04-5ad9-4da7-aee3-571710a3213d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7e6a04-5ad9-4da7-aee3-571710a3213d-maas-api-xks-ca\") pod \"3d7e6a04-5ad9-4da7-aee3-571710a3213d\" (UID: \"3d7e6a04-5ad9-4da7-aee3-571710a3213d\") " Nov 02 18:16:49.897627 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:49.897596 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7e6a04-5ad9-4da7-aee3-571710a3213d-maas-api-openshift-ca" pod "3d7e6a04-5ad9-4da7-aee3-571710a3213d" (UID: "3d7e6a04-5ad9-4da7-aee3-571710a3213d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:16:49.899882 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:49.899824 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7e6a04-5ad9-4da7-aee3-571710a3213d-kube-api-access-5j7sg" pod "3d7e6a04-5ad9-4da7-aee3-571710a3213d" (UID: "3d7e6a04-5ad9-4da7-aee3-571710a3213d"). InnerVolumeSpecName "kube-api-access-5j7sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:49.899882 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:49.899864 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d7e6a04-5ad9-4da7-aee3-571710a3213d-maas-api-xks-ca" pod "3d7e6a04-5ad9-4da7-aee3-571710a3213d" (UID: "3d7e6a04-5ad9-4da7-aee3-571710a3213d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:16:49.998791 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:49.998720 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j7sg\" (UniqueName: \"kubernetes.io/projected/3d7e6a04-5ad9-4da7-aee3-571710a3213d-kube-api-access-5j7sg\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:49.998791 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:49.998750 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7e6a04-5ad9-4da7-aee3-571710a3213d-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:49.998791 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:49.998760 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7e6a04-5ad9-4da7-aee3-571710a3213d-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:16:50.699949 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:50.699922 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm" Nov 02 18:16:50.699949 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:50.699943 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm" event={"ID":"3d7e6a04-5ad9-4da7-aee3-571710a3213d","Type":"ContainerDied","Data":"383a91cbbf0df27899630c634470bfd84e9f7449167ffb4bbf8edbcaf6e43c4e"} Nov 02 18:16:50.700168 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:16:50.699990 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="383a91cbbf0df27899630c634470bfd84e9f7449167ffb4bbf8edbcaf6e43c4e" Nov 02 18:16:51.769813 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:16:51.769775 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:17:02.896143 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:02.896053 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p"] Nov 02 18:17:02.896672 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:02.896650 4860 cpu_manager.go:414] "containerMap: removing container" podUID="3d7e6a04-5ad9-4da7-aee3-571710a3213d" containerName="revoke-keys" Nov 02 18:17:02.896763 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:02.896674 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7e6a04-5ad9-4da7-aee3-571710a3213d" containerName="revoke-keys" Nov 02 18:17:02.896835 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:02.896820 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7e6a04-5ad9-4da7-aee3-571710a3213d" containerName="revoke-keys" Nov 02 18:17:02.900271 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:02.900244 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:02.928917 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:02.928884 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p"] Nov 02 18:17:03.016233 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.016191 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nqkt\" (UniqueName: \"kubernetes.io/projected/48139446-b329-47dc-a692-9b9a89e26ed7-kube-api-access-7nqkt\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.016233 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.016234 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48139446-b329-47dc-a692-9b9a89e26ed7-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.016479 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.016346 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48139446-b329-47dc-a692-9b9a89e26ed7-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.016479 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.016388 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48139446-b329-47dc-a692-9b9a89e26ed7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.016479 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.016417 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48139446-b329-47dc-a692-9b9a89e26ed7-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.016479 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.016443 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48139446-b329-47dc-a692-9b9a89e26ed7-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.016644 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.016499 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48139446-b329-47dc-a692-9b9a89e26ed7-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.016644 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.016523 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48139446-b329-47dc-a692-9b9a89e26ed7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.016644 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.016547 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48139446-b329-47dc-a692-9b9a89e26ed7-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.117730 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.117685 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48139446-b329-47dc-a692-9b9a89e26ed7-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.117955 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.117757 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nqkt\" (UniqueName: \"kubernetes.io/projected/48139446-b329-47dc-a692-9b9a89e26ed7-kube-api-access-7nqkt\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.117955 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.117784 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48139446-b329-47dc-a692-9b9a89e26ed7-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.117955 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.117881 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48139446-b329-47dc-a692-9b9a89e26ed7-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.117955 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.117924 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48139446-b329-47dc-a692-9b9a89e26ed7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.118241 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.117962 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48139446-b329-47dc-a692-9b9a89e26ed7-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.118241 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.117985 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48139446-b329-47dc-a692-9b9a89e26ed7-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.118241 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.118037 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48139446-b329-47dc-a692-9b9a89e26ed7-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.118241 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.118096 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48139446-b329-47dc-a692-9b9a89e26ed7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.118241 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.118206 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48139446-b329-47dc-a692-9b9a89e26ed7-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.118241 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.118226 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48139446-b329-47dc-a692-9b9a89e26ed7-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.118577 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.118308 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48139446-b329-47dc-a692-9b9a89e26ed7-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.118577 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.118404 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48139446-b329-47dc-a692-9b9a89e26ed7-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.118815 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.118795 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48139446-b329-47dc-a692-9b9a89e26ed7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.120438 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.120416 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48139446-b329-47dc-a692-9b9a89e26ed7-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.120605 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.120585 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48139446-b329-47dc-a692-9b9a89e26ed7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.141553 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.141525 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48139446-b329-47dc-a692-9b9a89e26ed7-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.141881 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.141833 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nqkt\" (UniqueName: \"kubernetes.io/projected/48139446-b329-47dc-a692-9b9a89e26ed7-kube-api-access-7nqkt\") pod \"maas-default-gateway-openshift-default-76f4876667-6gl6p\" (UID: \"48139446-b329-47dc-a692-9b9a89e26ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.215035 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.214954 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:03.531342 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.531312 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p"] Nov 02 18:17:03.533064 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:17:03.533030 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48139446_b329_47dc_a692_9b9a89e26ed7.slice/crio-431d4d8b78722ff113f9482d2f0c4ceb65a61703197dfda727ea4372601983cd WatchSource:0}: Error finding container 431d4d8b78722ff113f9482d2f0c4ceb65a61703197dfda727ea4372601983cd: Status 404 returned error can't find the container with id 431d4d8b78722ff113f9482d2f0c4ceb65a61703197dfda727ea4372601983cd Nov 02 18:17:03.535129 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.535093 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:17:03.535235 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.535174 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:17:03.535307 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.535247 4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:17:03.776788 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.776747 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" event={"ID":"48139446-b329-47dc-a692-9b9a89e26ed7","Type":"ContainerStarted","Data":"bd2882f1b1b091c3981ac7bac1bf2b2be2a1a2eb1849ae0d127fe10203713e49"} Nov 02 18:17:03.777010 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.776874 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" event={"ID":"48139446-b329-47dc-a692-9b9a89e26ed7","Type":"ContainerStarted","Data":"431d4d8b78722ff113f9482d2f0c4ceb65a61703197dfda727ea4372601983cd"} Nov 02 18:17:03.823224 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:03.823158 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" podStartSLOduration=1.823143982 podStartE2EDuration="1.823143982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:03.822047326 +0000 UTC m=+2842.443407107" watchObservedRunningTime="2026-11-02 18:17:03.823143982 +0000 UTC m=+2842.444503757" Nov 02 18:17:04.215821 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:04.215723 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:04.217423 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:04.217391 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6gl6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 02 18:17:04.217563 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:04.217465 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" podUID="48139446-b329-47dc-a692-9b9a89e26ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 02 18:17:05.215581 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:05.215540 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6gl6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 02 18:17:05.215792 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:05.215627 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" podUID="48139446-b329-47dc-a692-9b9a89e26ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 02 18:17:06.216112 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:06.216072 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6gl6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 02 18:17:06.216553 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:06.216130 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" podUID="48139446-b329-47dc-a692-9b9a89e26ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 02 18:17:07.216348 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:07.216306 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6gl6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 02 18:17:07.216736 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:07.216384 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" podUID="48139446-b329-47dc-a692-9b9a89e26ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 02 18:17:08.216171 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:08.216133 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6gl6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 02 18:17:08.216381 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:08.216202 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" podUID="48139446-b329-47dc-a692-9b9a89e26ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 02 18:17:09.215747 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:09.215709 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6gl6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 02 18:17:09.215955 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:09.215762 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" podUID="48139446-b329-47dc-a692-9b9a89e26ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 02 18:17:10.216038 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:10.215997 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6gl6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 02 18:17:10.216436 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:10.216066 4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" podUID="48139446-b329-47dc-a692-9b9a89e26ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 02 18:17:11.719481 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:11.719441 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:11.719926 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:11.719806 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:11.720692 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:11.720669 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" Nov 02 18:17:21.772839 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:17:21.772791 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:17:46.467844 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:46.467805 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t"] Nov 02 18:17:46.471275 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:46.471256 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t" Nov 02 18:17:46.500522 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:46.500495 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:17:46.500678 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:46.500519 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 18:17:46.503463 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:46.503437 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t"] Nov 02 18:17:46.561769 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:46.561736 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf7xn\" (UniqueName: \"kubernetes.io/projected/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-kube-api-access-lf7xn\") pod \"maas-api-revoke-sub-c1e271a9-xqt8t\" (UID: \"ffcce8aa-3577-48f1-ab6d-e8a73a59eace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t" Nov 02 18:17:46.561991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:46.561778 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xqt8t\" (UID: \"ffcce8aa-3577-48f1-ab6d-e8a73a59eace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t" Nov 02 18:17:46.561991 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:46.561888 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xqt8t\" (UID: \"ffcce8aa-3577-48f1-ab6d-e8a73a59eace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t" Nov 02 18:17:46.663324 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:46.663292 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf7xn\" (UniqueName: \"kubernetes.io/projected/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-kube-api-access-lf7xn\") pod \"maas-api-revoke-sub-c1e271a9-xqt8t\" (UID: \"ffcce8aa-3577-48f1-ab6d-e8a73a59eace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t" Nov 02 18:17:46.663483 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:46.663329 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xqt8t\" (UID: \"ffcce8aa-3577-48f1-ab6d-e8a73a59eace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t" Nov 02 18:17:46.663483 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:46.663350 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xqt8t\" (UID: \"ffcce8aa-3577-48f1-ab6d-e8a73a59eace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t" Nov 02 18:17:46.663972 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:46.663951 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xqt8t\" (UID: \"ffcce8aa-3577-48f1-ab6d-e8a73a59eace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t" Nov 02 18:17:46.665623 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:46.665603 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xqt8t\" (UID: \"ffcce8aa-3577-48f1-ab6d-e8a73a59eace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t" Nov 02 18:17:46.686335 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:46.686313 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf7xn\" (UniqueName: \"kubernetes.io/projected/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-kube-api-access-lf7xn\") pod \"maas-api-revoke-sub-c1e271a9-xqt8t\" (UID: \"ffcce8aa-3577-48f1-ab6d-e8a73a59eace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t" Nov 02 18:17:46.784106 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:46.784027 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t" Nov 02 18:17:47.097871 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:47.097825 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t"] Nov 02 18:17:47.099619 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:17:47.099580 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcce8aa_3577_48f1_ab6d_e8a73a59eace.slice/crio-8890f68888df7d7138a0f322f87f635052b7b862e4b3b4a9600e2b2faaed51f0 WatchSource:0}: Error finding container 8890f68888df7d7138a0f322f87f635052b7b862e4b3b4a9600e2b2faaed51f0: Status 404 returned error can't find the container with id 8890f68888df7d7138a0f322f87f635052b7b862e4b3b4a9600e2b2faaed51f0 Nov 02 18:17:48.056983 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:48.056937 4860 generic.go:356] "Generic (PLEG): container finished" podID="ffcce8aa-3577-48f1-ab6d-e8a73a59eace" containerID="a93317c396de4e81d80bf55849d232ced6087141a0cafcab538489cade4b2949" exitCode=0 Nov 02 18:17:48.057379 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:48.057014 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t" event={"ID":"ffcce8aa-3577-48f1-ab6d-e8a73a59eace","Type":"ContainerDied","Data":"a93317c396de4e81d80bf55849d232ced6087141a0cafcab538489cade4b2949"} Nov 02 18:17:48.057379 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:48.057051 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t" event={"ID":"ffcce8aa-3577-48f1-ab6d-e8a73a59eace","Type":"ContainerStarted","Data":"8890f68888df7d7138a0f322f87f635052b7b862e4b3b4a9600e2b2faaed51f0"} Nov 02 18:17:49.212589 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:49.212565 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t" Nov 02 18:17:49.285362 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:49.285326 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-kube-api-access-lf7xn\" (UniqueName: \"kubernetes.io/projected/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-kube-api-access-lf7xn\") pod \"ffcce8aa-3577-48f1-ab6d-e8a73a59eace\" (UID: \"ffcce8aa-3577-48f1-ab6d-e8a73a59eace\") " Nov 02 18:17:49.285516 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:49.285402 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-maas-api-xks-ca\") pod \"ffcce8aa-3577-48f1-ab6d-e8a73a59eace\" (UID: \"ffcce8aa-3577-48f1-ab6d-e8a73a59eace\") " Nov 02 18:17:49.285516 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:49.285492 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-maas-api-openshift-ca\") pod \"ffcce8aa-3577-48f1-ab6d-e8a73a59eace\" (UID: \"ffcce8aa-3577-48f1-ab6d-e8a73a59eace\") " Nov 02 18:17:49.285909 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:49.285875 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-maas-api-openshift-ca" pod "ffcce8aa-3577-48f1-ab6d-e8a73a59eace" (UID: "ffcce8aa-3577-48f1-ab6d-e8a73a59eace"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:17:49.287521 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:49.287492 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-maas-api-xks-ca" pod "ffcce8aa-3577-48f1-ab6d-e8a73a59eace" (UID: "ffcce8aa-3577-48f1-ab6d-e8a73a59eace"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:17:49.287621 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:49.287495 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-kube-api-access-lf7xn" pod "ffcce8aa-3577-48f1-ab6d-e8a73a59eace" (UID: "ffcce8aa-3577-48f1-ab6d-e8a73a59eace"). InnerVolumeSpecName "kube-api-access-lf7xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:17:49.387145 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:49.387071 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf7xn\" (UniqueName: \"kubernetes.io/projected/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-kube-api-access-lf7xn\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:17:49.387145 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:49.387098 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:17:49.387145 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:49.387108 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffcce8aa-3577-48f1-ab6d-e8a73a59eace-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:17:50.074257 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:50.074226 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t" Nov 02 18:17:50.074431 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:50.074222 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t" event={"ID":"ffcce8aa-3577-48f1-ab6d-e8a73a59eace","Type":"ContainerDied","Data":"8890f68888df7d7138a0f322f87f635052b7b862e4b3b4a9600e2b2faaed51f0"} Nov 02 18:17:50.074431 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:17:50.074350 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8890f68888df7d7138a0f322f87f635052b7b862e4b3b4a9600e2b2faaed51f0" Nov 02 18:17:51.769465 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:17:51.769426 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:18:21.769967 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:18:21.769927 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:18:51.769797 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:18:51.769758 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:19:21.769615 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:19:21.769574 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:19:42.140760 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:19:42.140730 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 18:19:42.153277 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:19:42.153250 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 18:19:42.155370 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:19:42.155347 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 18:19:42.167698 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:19:42.167679 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 18:19:51.769800 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:19:51.769747 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:20:00.523137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:20:00.523025 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-w4772 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:20:00.523769 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:20:00.523729 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-w4772" podUID="94044c36-9d36-41a2-801c-afb684fdfd3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:20:21.769376 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:20:21.769335 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:20:51.769151 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:20:51.769114 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:20:59.032631 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:20:59.032591 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj"] Nov 02 18:20:59.044884 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:20:59.044845 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-232313e4-95zzj"] Nov 02 18:20:59.498690 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:20:59.498650 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08d029a4-bde0-4ff2-8d27-fa04d90dd1d1" path="/var/lib/kubelet/pods/08d029a4-bde0-4ff2-8d27-fa04d90dd1d1/volumes" Nov 02 18:21:21.769469 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:21:21.769421 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:21:50.032084 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:21:50.032049 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm"] Nov 02 18:21:50.044181 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:21:50.044152 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rhmhm"] Nov 02 18:21:51.497973 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:21:51.497939 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d7e6a04-5ad9-4da7-aee3-571710a3213d" path="/var/lib/kubelet/pods/3d7e6a04-5ad9-4da7-aee3-571710a3213d/volumes" Nov 02 18:21:51.769453 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:21:51.769371 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:22:21.769648 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:22:21.769606 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:22:28.299398 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.299358 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw"] Nov 02 18:22:28.300091 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.300067 4860 cpu_manager.go:414] "containerMap: removing container" podUID="ffcce8aa-3577-48f1-ab6d-e8a73a59eace" containerName="revoke-keys" Nov 02 18:22:28.300189 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.300093 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffcce8aa-3577-48f1-ab6d-e8a73a59eace" containerName="revoke-keys" Nov 02 18:22:28.300251 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.300230 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffcce8aa-3577-48f1-ab6d-e8a73a59eace" containerName="revoke-keys" Nov 02 18:22:28.303945 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.303909 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.319550 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.319520 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ce839a\"/\"openshift-service-ca.crt\"" Nov 02 18:22:28.319550 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.319559 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ce839a\"/\"kube-root-ca.crt\"" Nov 02 18:22:28.319741 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.319715 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ce839a\"/\"simulator-single-main-ce839a-kserve-self-signed-certs\"" Nov 02 18:22:28.319785 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.319745 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ce839a\"/\"default-dockercfg-98pcv\"" Nov 02 18:22:28.321830 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.321804 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw"] Nov 02 18:22:28.322118 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.322091 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-home\") pod \"simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.322333 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.322308 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-tmp-dir\") pod \"simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.322428 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.322349 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cblbd\" (UniqueName: \"kubernetes.io/projected/a108046b-f458-43e3-bceb-f208b7c468f5-kube-api-access-cblbd\") pod \"simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.322472 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.322421 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-dshm\") pod \"simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.322531 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.322510 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-model-cache\") pod \"simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.322613 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.322593 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a108046b-f458-43e3-bceb-f208b7c468f5-tls-certs\") pod \"simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.423455 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.423418 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-tmp-dir\") pod \"simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.423455 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.423460 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cblbd\" (UniqueName: \"kubernetes.io/projected/a108046b-f458-43e3-bceb-f208b7c468f5-kube-api-access-cblbd\") pod \"simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.423704 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.423476 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-dshm\") pod \"simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.423704 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.423508 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-model-cache\") pod \"simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.423704 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.423573 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a108046b-f458-43e3-bceb-f208b7c468f5-tls-certs\") pod \"simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.423704 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.423659 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-home\") pod \"simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.423992 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.423938 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-tmp-dir\") pod \"simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.424066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.423989 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-model-cache\") pod \"simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.424066 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.424022 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-home\") pod \"simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.426108 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.426083 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-dshm\") pod \"simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.426239 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.426149 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a108046b-f458-43e3-bceb-f208b7c468f5-tls-certs\") pod \"simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.446391 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.446354 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cblbd\" (UniqueName: \"kubernetes.io/projected/a108046b-f458-43e3-bceb-f208b7c468f5-kube-api-access-cblbd\") pod \"simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.617794 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.617758 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:28.917528 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:28.917502 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw"] Nov 02 18:22:28.918915 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:22:28.918887 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda108046b_f458_43e3_bceb_f208b7c468f5.slice/crio-e1a56ec8dc787b87e63f4dd851ac3f94d8e1810f28dacb657676a67f7153c675 WatchSource:0}: Error finding container e1a56ec8dc787b87e63f4dd851ac3f94d8e1810f28dacb657676a67f7153c675: Status 404 returned error can't find the container with id e1a56ec8dc787b87e63f4dd851ac3f94d8e1810f28dacb657676a67f7153c675 Nov 02 18:22:29.002453 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:29.002378 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" event={"ID":"a108046b-f458-43e3-bceb-f208b7c468f5","Type":"ContainerStarted","Data":"a02f44a9cd01dcdeb69da658951a75fdf1c24577a1ae6065a3e3771a6d9dc9da"} Nov 02 18:22:29.002668 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:29.002468 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" event={"ID":"a108046b-f458-43e3-bceb-f208b7c468f5","Type":"ContainerStarted","Data":"e1a56ec8dc787b87e63f4dd851ac3f94d8e1810f28dacb657676a67f7153c675"} Nov 02 18:22:29.045321 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:29.045263 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" podStartSLOduration=1.04524521 podStartE2EDuration="1.04524521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:22:29.044738152 +0000 UTC m=+3167.666097929" watchObservedRunningTime="2026-11-02 18:22:29.04524521 +0000 UTC m=+3167.666604988" Nov 02 18:22:29.618133 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:29.618102 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:38.618316 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:38.618280 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:38.631303 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:38.631270 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:39.075332 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:39.075300 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:22:44.502034 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:44.501995 4860 scope.go:122] "RemoveContainer" containerID="06c9172e450733f95dbc9624a7521bfed9758bc70b0d58ba34c2ff6a73263327" Nov 02 18:22:50.051975 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:50.051937 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t"] Nov 02 18:22:50.063452 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:50.063423 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xqt8t"] Nov 02 18:22:51.497328 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:22:51.497292 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffcce8aa-3577-48f1-ab6d-e8a73a59eace" path="/var/lib/kubelet/pods/ffcce8aa-3577-48f1-ab6d-e8a73a59eace/volumes" Nov 02 18:22:51.770025 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:22:51.769927 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:23:00.373735 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:00.373646 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp"] Nov 02 18:23:00.379248 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:00.379223 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp" Nov 02 18:23:00.396254 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:00.396229 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp"] Nov 02 18:23:00.405887 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:00.405833 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:23:00.405993 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:00.405901 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 18:23:00.531821 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:00.531771 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jshm\" (UniqueName: \"kubernetes.io/projected/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-kube-api-access-9jshm\") pod \"maas-api-revoke-sub-908b634a-8hfcp\" (UID: \"4846f6c7-b7c8-4ced-bb3b-3ef9888fac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp" Nov 02 18:23:00.532008 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:00.531840 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-908b634a-8hfcp\" (UID: \"4846f6c7-b7c8-4ced-bb3b-3ef9888fac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp" Nov 02 18:23:00.532008 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:00.531906 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-908b634a-8hfcp\" (UID: \"4846f6c7-b7c8-4ced-bb3b-3ef9888fac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp" Nov 02 18:23:00.632576 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:00.632472 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-908b634a-8hfcp\" (UID: \"4846f6c7-b7c8-4ced-bb3b-3ef9888fac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp" Nov 02 18:23:00.632576 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:00.632523 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-908b634a-8hfcp\" (UID: \"4846f6c7-b7c8-4ced-bb3b-3ef9888fac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp" Nov 02 18:23:00.632831 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:00.632607 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jshm\" (UniqueName: \"kubernetes.io/projected/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-kube-api-access-9jshm\") pod \"maas-api-revoke-sub-908b634a-8hfcp\" (UID: \"4846f6c7-b7c8-4ced-bb3b-3ef9888fac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp" Nov 02 18:23:00.633274 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:00.633248 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-908b634a-8hfcp\" (UID: \"4846f6c7-b7c8-4ced-bb3b-3ef9888fac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp" Nov 02 18:23:00.634960 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:00.634939 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-908b634a-8hfcp\" (UID: \"4846f6c7-b7c8-4ced-bb3b-3ef9888fac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp" Nov 02 18:23:00.656111 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:00.656082 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jshm\" (UniqueName: \"kubernetes.io/projected/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-kube-api-access-9jshm\") pod \"maas-api-revoke-sub-908b634a-8hfcp\" (UID: \"4846f6c7-b7c8-4ced-bb3b-3ef9888fac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp" Nov 02 18:23:00.691964 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:00.691941 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp" Nov 02 18:23:00.991155 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:00.991117 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp"] Nov 02 18:23:00.992565 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:23:00.992532 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4846f6c7_b7c8_4ced_bb3b_3ef9888fac93.slice/crio-2c20fe89aa00747f318f3ce59f70c52e0d4c84a1d7b6e17fa6a6c7f967873e25 WatchSource:0}: Error finding container 2c20fe89aa00747f318f3ce59f70c52e0d4c84a1d7b6e17fa6a6c7f967873e25: Status 404 returned error can't find the container with id 2c20fe89aa00747f318f3ce59f70c52e0d4c84a1d7b6e17fa6a6c7f967873e25 Nov 02 18:23:01.210249 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:01.210206 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp" event={"ID":"4846f6c7-b7c8-4ced-bb3b-3ef9888fac93","Type":"ContainerStarted","Data":"ac9affff94fe6e60b60fbbcc9930c4a8c7e8bbb015d0b1cac646da28c4e6582d"} Nov 02 18:23:01.210412 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:01.210287 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp" event={"ID":"4846f6c7-b7c8-4ced-bb3b-3ef9888fac93","Type":"ContainerStarted","Data":"2c20fe89aa00747f318f3ce59f70c52e0d4c84a1d7b6e17fa6a6c7f967873e25"} Nov 02 18:23:01.250961 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:01.250839 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp" podStartSLOduration=1.250827517 podStartE2EDuration="1.250827517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:23:01.250090699 +0000 UTC m=+3199.871450474" watchObservedRunningTime="2026-11-02 18:23:01.250827517 +0000 UTC m=+3199.872187293" Nov 02 18:23:02.217310 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:02.217273 4860 generic.go:356] "Generic (PLEG): container finished" podID="4846f6c7-b7c8-4ced-bb3b-3ef9888fac93" containerID="ac9affff94fe6e60b60fbbcc9930c4a8c7e8bbb015d0b1cac646da28c4e6582d" exitCode=0 Nov 02 18:23:02.217758 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:02.217331 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp" event={"ID":"4846f6c7-b7c8-4ced-bb3b-3ef9888fac93","Type":"ContainerDied","Data":"ac9affff94fe6e60b60fbbcc9930c4a8c7e8bbb015d0b1cac646da28c4e6582d"} Nov 02 18:23:03.364043 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:03.364018 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp" Nov 02 18:23:03.459615 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:03.459580 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-maas-api-xks-ca\") pod \"4846f6c7-b7c8-4ced-bb3b-3ef9888fac93\" (UID: \"4846f6c7-b7c8-4ced-bb3b-3ef9888fac93\") " Nov 02 18:23:03.459763 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:03.459661 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-kube-api-access-9jshm\" (UniqueName: \"kubernetes.io/projected/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-kube-api-access-9jshm\") pod \"4846f6c7-b7c8-4ced-bb3b-3ef9888fac93\" (UID: \"4846f6c7-b7c8-4ced-bb3b-3ef9888fac93\") " Nov 02 18:23:03.459763 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:03.459731 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-maas-api-openshift-ca\") pod \"4846f6c7-b7c8-4ced-bb3b-3ef9888fac93\" (UID: \"4846f6c7-b7c8-4ced-bb3b-3ef9888fac93\") " Nov 02 18:23:03.460118 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:03.460086 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-maas-api-openshift-ca" pod "4846f6c7-b7c8-4ced-bb3b-3ef9888fac93" (UID: "4846f6c7-b7c8-4ced-bb3b-3ef9888fac93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:23:03.461793 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:03.461773 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-maas-api-xks-ca" pod "4846f6c7-b7c8-4ced-bb3b-3ef9888fac93" (UID: "4846f6c7-b7c8-4ced-bb3b-3ef9888fac93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:23:03.461886 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:03.461795 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-kube-api-access-9jshm" pod "4846f6c7-b7c8-4ced-bb3b-3ef9888fac93" (UID: "4846f6c7-b7c8-4ced-bb3b-3ef9888fac93"). InnerVolumeSpecName "kube-api-access-9jshm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:03.560840 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:03.560810 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:23:03.560840 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:03.560837 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jshm\" (UniqueName: \"kubernetes.io/projected/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-kube-api-access-9jshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:23:03.561094 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:03.560869 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:23:04.231461 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:04.231427 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp" Nov 02 18:23:04.231674 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:04.231469 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp" event={"ID":"4846f6c7-b7c8-4ced-bb3b-3ef9888fac93","Type":"ContainerDied","Data":"2c20fe89aa00747f318f3ce59f70c52e0d4c84a1d7b6e17fa6a6c7f967873e25"} Nov 02 18:23:04.231674 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:04.231523 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c20fe89aa00747f318f3ce59f70c52e0d4c84a1d7b6e17fa6a6c7f967873e25" Nov 02 18:23:11.885245 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:11.885202 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9"] Nov 02 18:23:11.885896 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:11.885868 4860 cpu_manager.go:414] "containerMap: removing container" podUID="4846f6c7-b7c8-4ced-bb3b-3ef9888fac93" containerName="revoke-keys" Nov 02 18:23:11.885896 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:11.885894 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4846f6c7-b7c8-4ced-bb3b-3ef9888fac93" containerName="revoke-keys" Nov 02 18:23:11.886148 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:11.886023 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4846f6c7-b7c8-4ced-bb3b-3ef9888fac93" containerName="revoke-keys" Nov 02 18:23:11.890151 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:11.890121 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9" Nov 02 18:23:11.910704 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:11.910675 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9"] Nov 02 18:23:11.919420 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:11.919395 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:23:11.919567 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:11.919501 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 18:23:11.940526 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:11.940496 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb6r9\" (UniqueName: \"kubernetes.io/projected/74527410-7326-4924-bb29-a15bf63ec6bc-kube-api-access-hb6r9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9\" (UID: \"74527410-7326-4924-bb29-a15bf63ec6bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9" Nov 02 18:23:11.940650 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:11.940553 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74527410-7326-4924-bb29-a15bf63ec6bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9\" (UID: \"74527410-7326-4924-bb29-a15bf63ec6bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9" Nov 02 18:23:11.940704 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:11.940673 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74527410-7326-4924-bb29-a15bf63ec6bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9\" (UID: \"74527410-7326-4924-bb29-a15bf63ec6bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9" Nov 02 18:23:12.041351 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:12.041310 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb6r9\" (UniqueName: \"kubernetes.io/projected/74527410-7326-4924-bb29-a15bf63ec6bc-kube-api-access-hb6r9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9\" (UID: \"74527410-7326-4924-bb29-a15bf63ec6bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9" Nov 02 18:23:12.041545 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:12.041379 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74527410-7326-4924-bb29-a15bf63ec6bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9\" (UID: \"74527410-7326-4924-bb29-a15bf63ec6bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9" Nov 02 18:23:12.041545 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:12.041406 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74527410-7326-4924-bb29-a15bf63ec6bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9\" (UID: \"74527410-7326-4924-bb29-a15bf63ec6bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9" Nov 02 18:23:12.042118 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:12.042090 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74527410-7326-4924-bb29-a15bf63ec6bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9\" (UID: \"74527410-7326-4924-bb29-a15bf63ec6bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9" Nov 02 18:23:12.043810 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:12.043782 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74527410-7326-4924-bb29-a15bf63ec6bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9\" (UID: \"74527410-7326-4924-bb29-a15bf63ec6bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9" Nov 02 18:23:12.064654 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:12.064626 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb6r9\" (UniqueName: \"kubernetes.io/projected/74527410-7326-4924-bb29-a15bf63ec6bc-kube-api-access-hb6r9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9\" (UID: \"74527410-7326-4924-bb29-a15bf63ec6bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9" Nov 02 18:23:12.205272 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:12.205188 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9" Nov 02 18:23:12.504817 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:12.504779 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9"] Nov 02 18:23:12.506153 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:23:12.506121 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74527410_7326_4924_bb29_a15bf63ec6bc.slice/crio-6c6827d12f0820a0bdfd101c86d699756af6cb0ec6ab6001b04d76de98d07e0a WatchSource:0}: Error finding container 6c6827d12f0820a0bdfd101c86d699756af6cb0ec6ab6001b04d76de98d07e0a: Status 404 returned error can't find the container with id 6c6827d12f0820a0bdfd101c86d699756af6cb0ec6ab6001b04d76de98d07e0a Nov 02 18:23:13.285276 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:13.285248 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9" event={"ID":"74527410-7326-4924-bb29-a15bf63ec6bc","Type":"ContainerStarted","Data":"c21a80badc5e0ae32696841531bd872484ff7f1b551e2f1866de8bfd7f053d79"} Nov 02 18:23:13.285640 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:13.285296 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9" event={"ID":"74527410-7326-4924-bb29-a15bf63ec6bc","Type":"ContainerStarted","Data":"6c6827d12f0820a0bdfd101c86d699756af6cb0ec6ab6001b04d76de98d07e0a"} Nov 02 18:23:13.325268 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:13.325218 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9" podStartSLOduration=2.325204481 podStartE2EDuration="2.325204481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:23:13.324919699 +0000 UTC m=+3211.946279474" watchObservedRunningTime="2026-11-02 18:23:13.325204481 +0000 UTC m=+3211.946564256" Nov 02 18:23:14.293560 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:14.293520 4860 generic.go:356] "Generic (PLEG): container finished" podID="74527410-7326-4924-bb29-a15bf63ec6bc" containerID="c21a80badc5e0ae32696841531bd872484ff7f1b551e2f1866de8bfd7f053d79" exitCode=0 Nov 02 18:23:14.294003 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:14.293608 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9" event={"ID":"74527410-7326-4924-bb29-a15bf63ec6bc","Type":"ContainerDied","Data":"c21a80badc5e0ae32696841531bd872484ff7f1b551e2f1866de8bfd7f053d79"} Nov 02 18:23:15.443032 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:15.442999 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9" Nov 02 18:23:15.472440 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:15.472413 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74527410-7326-4924-bb29-a15bf63ec6bc-kube-api-access-hb6r9\" (UniqueName: \"kubernetes.io/projected/74527410-7326-4924-bb29-a15bf63ec6bc-kube-api-access-hb6r9\") pod \"74527410-7326-4924-bb29-a15bf63ec6bc\" (UID: \"74527410-7326-4924-bb29-a15bf63ec6bc\") " Nov 02 18:23:15.472563 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:15.472482 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74527410-7326-4924-bb29-a15bf63ec6bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74527410-7326-4924-bb29-a15bf63ec6bc-maas-api-openshift-ca\") pod \"74527410-7326-4924-bb29-a15bf63ec6bc\" (UID: \"74527410-7326-4924-bb29-a15bf63ec6bc\") " Nov 02 18:23:15.472563 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:15.472532 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74527410-7326-4924-bb29-a15bf63ec6bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74527410-7326-4924-bb29-a15bf63ec6bc-maas-api-xks-ca\") pod \"74527410-7326-4924-bb29-a15bf63ec6bc\" (UID: \"74527410-7326-4924-bb29-a15bf63ec6bc\") " Nov 02 18:23:15.472911 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:15.472885 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74527410-7326-4924-bb29-a15bf63ec6bc-maas-api-openshift-ca" pod "74527410-7326-4924-bb29-a15bf63ec6bc" (UID: "74527410-7326-4924-bb29-a15bf63ec6bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:23:15.474570 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:15.474546 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74527410-7326-4924-bb29-a15bf63ec6bc-maas-api-xks-ca" pod "74527410-7326-4924-bb29-a15bf63ec6bc" (UID: "74527410-7326-4924-bb29-a15bf63ec6bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:23:15.474643 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:15.474623 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74527410-7326-4924-bb29-a15bf63ec6bc-kube-api-access-hb6r9" pod "74527410-7326-4924-bb29-a15bf63ec6bc" (UID: "74527410-7326-4924-bb29-a15bf63ec6bc"). InnerVolumeSpecName "kube-api-access-hb6r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:15.510513 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:23:15.510479 4860 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74527410_7326_4924_bb29_a15bf63ec6bc.slice\": RecentStats: unable to find data in memory cache]" Nov 02 18:23:15.574339 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:15.574308 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb6r9\" (UniqueName: \"kubernetes.io/projected/74527410-7326-4924-bb29-a15bf63ec6bc-kube-api-access-hb6r9\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:23:15.574339 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:15.574336 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74527410-7326-4924-bb29-a15bf63ec6bc-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:23:15.574571 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:15.574353 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74527410-7326-4924-bb29-a15bf63ec6bc-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:23:16.307502 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:16.307467 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9" Nov 02 18:23:16.307677 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:16.307496 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9" event={"ID":"74527410-7326-4924-bb29-a15bf63ec6bc","Type":"ContainerDied","Data":"6c6827d12f0820a0bdfd101c86d699756af6cb0ec6ab6001b04d76de98d07e0a"} Nov 02 18:23:16.307677 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:16.307552 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c6827d12f0820a0bdfd101c86d699756af6cb0ec6ab6001b04d76de98d07e0a" Nov 02 18:23:16.584059 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:16.583962 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw"] Nov 02 18:23:21.770016 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:23:21.769975 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:23:25.676041 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:25.676005 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9"] Nov 02 18:23:25.688407 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:25.688374 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ce839a-cc6s9"] Nov 02 18:23:27.497423 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:27.497388 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74527410-7326-4924-bb29-a15bf63ec6bc" path="/var/lib/kubelet/pods/74527410-7326-4924-bb29-a15bf63ec6bc/volumes" Nov 02 18:23:31.604339 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:31.604294 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" podUID="a108046b-f458-43e3-bceb-f208b7c468f5" containerName="main" containerID="cri-o://a02f44a9cd01dcdeb69da658951a75fdf1c24577a1ae6065a3e3771a6d9dc9da" gracePeriod=45 Nov 02 18:23:31.864235 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:31.864175 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:23:31.924674 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:31.924642 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a108046b-f458-43e3-bceb-f208b7c468f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/a108046b-f458-43e3-bceb-f208b7c468f5-tls-certs\") pod \"a108046b-f458-43e3-bceb-f208b7c468f5\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " Nov 02 18:23:31.924858 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:31.924731 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-tmp-dir\") pod \"a108046b-f458-43e3-bceb-f208b7c468f5\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " Nov 02 18:23:31.924858 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:31.924759 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-home\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-home\") pod \"a108046b-f458-43e3-bceb-f208b7c468f5\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " Nov 02 18:23:31.924858 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:31.924778 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-dshm\") pod \"a108046b-f458-43e3-bceb-f208b7c468f5\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " Nov 02 18:23:31.924858 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:31.924820 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-model-cache\") pod \"a108046b-f458-43e3-bceb-f208b7c468f5\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " Nov 02 18:23:31.925100 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:31.925060 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a108046b-f458-43e3-bceb-f208b7c468f5-kube-api-access-cblbd\" (UniqueName: \"kubernetes.io/projected/a108046b-f458-43e3-bceb-f208b7c468f5-kube-api-access-cblbd\") pod \"a108046b-f458-43e3-bceb-f208b7c468f5\" (UID: \"a108046b-f458-43e3-bceb-f208b7c468f5\") " Nov 02 18:23:31.925228 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:31.925063 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-tmp-dir" pod "a108046b-f458-43e3-bceb-f208b7c468f5" (UID: "a108046b-f458-43e3-bceb-f208b7c468f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:31.925296 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:31.925155 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-home" pod "a108046b-f458-43e3-bceb-f208b7c468f5" (UID: "a108046b-f458-43e3-bceb-f208b7c468f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:31.925444 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:31.925421 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-model-cache" pod "a108046b-f458-43e3-bceb-f208b7c468f5" (UID: "a108046b-f458-43e3-bceb-f208b7c468f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:31.925551 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:31.925521 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:23:31.925551 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:31.925546 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:23:31.925644 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:31.925560 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:23:31.927176 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:31.927152 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-dshm" pod "a108046b-f458-43e3-bceb-f208b7c468f5" (UID: "a108046b-f458-43e3-bceb-f208b7c468f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:23:31.927438 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:31.927410 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a108046b-f458-43e3-bceb-f208b7c468f5-tls-certs" pod "a108046b-f458-43e3-bceb-f208b7c468f5" (UID: "a108046b-f458-43e3-bceb-f208b7c468f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:23:31.927536 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:31.927432 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a108046b-f458-43e3-bceb-f208b7c468f5-kube-api-access-cblbd" pod "a108046b-f458-43e3-bceb-f208b7c468f5" (UID: "a108046b-f458-43e3-bceb-f208b7c468f5"). InnerVolumeSpecName "kube-api-access-cblbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:23:32.027043 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:32.026994 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cblbd\" (UniqueName: \"kubernetes.io/projected/a108046b-f458-43e3-bceb-f208b7c468f5-kube-api-access-cblbd\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:23:32.027043 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:32.027037 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a108046b-f458-43e3-bceb-f208b7c468f5-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:23:32.027043 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:32.027047 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a108046b-f458-43e3-bceb-f208b7c468f5-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:23:32.404083 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:32.404050 4860 generic.go:356] "Generic (PLEG): container finished" podID="a108046b-f458-43e3-bceb-f208b7c468f5" containerID="a02f44a9cd01dcdeb69da658951a75fdf1c24577a1ae6065a3e3771a6d9dc9da" exitCode=0 Nov 02 18:23:32.404278 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:32.404101 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" Nov 02 18:23:32.404278 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:32.404140 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" event={"ID":"a108046b-f458-43e3-bceb-f208b7c468f5","Type":"ContainerDied","Data":"a02f44a9cd01dcdeb69da658951a75fdf1c24577a1ae6065a3e3771a6d9dc9da"} Nov 02 18:23:32.404278 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:32.404186 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw" event={"ID":"a108046b-f458-43e3-bceb-f208b7c468f5","Type":"ContainerDied","Data":"e1a56ec8dc787b87e63f4dd851ac3f94d8e1810f28dacb657676a67f7153c675"} Nov 02 18:23:32.404278 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:32.404205 4860 scope.go:122] "RemoveContainer" containerID="a02f44a9cd01dcdeb69da658951a75fdf1c24577a1ae6065a3e3771a6d9dc9da" Nov 02 18:23:32.417228 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:32.417164 4860 scope.go:122] "RemoveContainer" containerID="a02f44a9cd01dcdeb69da658951a75fdf1c24577a1ae6065a3e3771a6d9dc9da" Nov 02 18:23:32.417530 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:23:32.417506 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02f44a9cd01dcdeb69da658951a75fdf1c24577a1ae6065a3e3771a6d9dc9da\": container with ID starting with a02f44a9cd01dcdeb69da658951a75fdf1c24577a1ae6065a3e3771a6d9dc9da not found: ID does not exist" containerID="a02f44a9cd01dcdeb69da658951a75fdf1c24577a1ae6065a3e3771a6d9dc9da" Nov 02 18:23:32.417600 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:32.417543 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02f44a9cd01dcdeb69da658951a75fdf1c24577a1ae6065a3e3771a6d9dc9da"} err="failed to get container status \"a02f44a9cd01dcdeb69da658951a75fdf1c24577a1ae6065a3e3771a6d9dc9da\": rpc error: code = NotFound desc = could not find container \"a02f44a9cd01dcdeb69da658951a75fdf1c24577a1ae6065a3e3771a6d9dc9da\": container with ID starting with a02f44a9cd01dcdeb69da658951a75fdf1c24577a1ae6065a3e3771a6d9dc9da not found: ID does not exist" Nov 02 18:23:32.457796 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:32.457753 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw"] Nov 02 18:23:32.463615 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:32.463588 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-ce839a/simulator-single-main-ce839a-kserve-55b5f7c8f-f5qtw"] Nov 02 18:23:33.498001 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:33.497965 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a108046b-f458-43e3-bceb-f208b7c468f5" path="/var/lib/kubelet/pods/a108046b-f458-43e3-bceb-f208b7c468f5/volumes" Nov 02 18:23:44.524861 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:23:44.524824 4860 scope.go:122] "RemoveContainer" containerID="79a37f25a1c526bf14c3a231eb4b26b2753b045b86edb7f557008e97dc9b11ff" Nov 02 18:23:51.770054 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:23:51.770016 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:24:21.769425 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:24:21.769384 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:24:42.184215 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:24:42.184187 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 18:24:42.196993 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:24:42.196968 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 18:24:42.199332 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:24:42.199307 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 18:24:42.215815 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:24:42.215786 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 18:24:44.552318 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:24:44.552287 4860 scope.go:122] "RemoveContainer" containerID="a93317c396de4e81d80bf55849d232ced6087141a0cafcab538489cade4b2949" Nov 02 18:24:51.769208 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:24:51.769169 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:25:21.769967 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:25:21.769927 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:25:51.769887 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:25:51.769759 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:26:13.504554 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.504480 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg"] Nov 02 18:26:13.504956 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.504921 4860 cpu_manager.go:414] "containerMap: removing container" podUID="a108046b-f458-43e3-bceb-f208b7c468f5" containerName="main" Nov 02 18:26:13.504956 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.504935 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a108046b-f458-43e3-bceb-f208b7c468f5" containerName="main" Nov 02 18:26:13.504956 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.504946 4860 cpu_manager.go:414] "containerMap: removing container" podUID="74527410-7326-4924-bb29-a15bf63ec6bc" containerName="revoke-keys" Nov 02 18:26:13.504956 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.504952 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74527410-7326-4924-bb29-a15bf63ec6bc" containerName="revoke-keys" Nov 02 18:26:13.505100 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.505041 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a108046b-f458-43e3-bceb-f208b7c468f5" containerName="main" Nov 02 18:26:13.505100 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.505049 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="74527410-7326-4924-bb29-a15bf63ec6bc" containerName="revoke-keys" Nov 02 18:26:13.508329 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.508309 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:13.554468 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.554409 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-dbe6df\"/\"simulator-single-main-dbe6df-kserve-self-signed-certs\"" Nov 02 18:26:13.554468 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.554426 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-dbe6df\"/\"kube-root-ca.crt\"" Nov 02 18:26:13.554665 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.554490 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-dbe6df\"/\"default-dockercfg-5cbrb\"" Nov 02 18:26:13.554665 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.554515 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-dbe6df\"/\"openshift-service-ca.crt\"" Nov 02 18:26:13.564883 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.564834 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg"] Nov 02 18:26:13.690989 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.690955 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4fd5039-5514-4e23-b080-f5495484504a-tls-certs\") pod \"simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:13.691183 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.691053 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdxwg\" (UniqueName: \"kubernetes.io/projected/c4fd5039-5514-4e23-b080-f5495484504a-kube-api-access-rdxwg\") pod \"simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:13.691232 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.691179 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-dshm\") pod \"simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:13.691272 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.691239 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-home\") pod \"simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:13.691337 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.691321 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-model-cache\") pod \"simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:13.691406 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.691346 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-tmp-dir\") pod \"simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:13.792562 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.792477 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-home\") pod \"simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:13.792562 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.792533 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-model-cache\") pod \"simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:13.792562 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.792554 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-tmp-dir\") pod \"simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:13.792903 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.792581 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4fd5039-5514-4e23-b080-f5495484504a-tls-certs\") pod \"simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:13.792903 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.792608 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdxwg\" (UniqueName: \"kubernetes.io/projected/c4fd5039-5514-4e23-b080-f5495484504a-kube-api-access-rdxwg\") pod \"simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:13.792903 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.792695 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-dshm\") pod \"simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:13.793095 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.792998 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-home\") pod \"simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:13.793095 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.793016 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-tmp-dir\") pod \"simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:13.793095 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.793068 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-model-cache\") pod \"simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:13.795062 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.795031 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-dshm\") pod \"simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:13.795371 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.795349 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4fd5039-5514-4e23-b080-f5495484504a-tls-certs\") pod \"simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:13.815435 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.815411 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdxwg\" (UniqueName: \"kubernetes.io/projected/c4fd5039-5514-4e23-b080-f5495484504a-kube-api-access-rdxwg\") pod \"simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:13.820257 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:13.820230 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:14.120927 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:14.120901 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg"] Nov 02 18:26:14.122644 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:26:14.122612 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4fd5039_5514_4e23_b080_f5495484504a.slice/crio-1795fb4ffad71786aa32b71402c550d1a0125c22a225f03e361a991b27ee8f0f WatchSource:0}: Error finding container 1795fb4ffad71786aa32b71402c550d1a0125c22a225f03e361a991b27ee8f0f: Status 404 returned error can't find the container with id 1795fb4ffad71786aa32b71402c550d1a0125c22a225f03e361a991b27ee8f0f Nov 02 18:26:14.364321 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:14.364279 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" event={"ID":"c4fd5039-5514-4e23-b080-f5495484504a","Type":"ContainerStarted","Data":"b966b88bab7f2cf504bddae6d5fd053ab14074539d9333f002659549eb198db1"} Nov 02 18:26:14.364527 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:14.364333 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" event={"ID":"c4fd5039-5514-4e23-b080-f5495484504a","Type":"ContainerStarted","Data":"1795fb4ffad71786aa32b71402c550d1a0125c22a225f03e361a991b27ee8f0f"} Nov 02 18:26:14.405732 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:14.405638 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" podStartSLOduration=1.405624875 podStartE2EDuration="1.405624875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:26:14.405003889 +0000 UTC m=+3393.026363662" watchObservedRunningTime="2026-11-02 18:26:14.405624875 +0000 UTC m=+3393.026984658" Nov 02 18:26:14.821359 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:14.821328 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:21.768268 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:26:21.768229 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:26:23.820736 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:23.820696 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:23.833664 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:23.833632 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:24.434634 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:24.434598 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:26:44.013775 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:44.013734 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5"] Nov 02 18:26:44.017910 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:44.017889 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5" Nov 02 18:26:44.033320 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:44.033292 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:26:44.033759 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:44.033410 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 18:26:44.036979 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:44.036952 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5"] Nov 02 18:26:44.181955 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:44.181913 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fdf0956-2383-4a6c-a5a2-c62c38854104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4f5d22e-scnr5\" (UID: \"4fdf0956-2383-4a6c-a5a2-c62c38854104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5" Nov 02 18:26:44.181955 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:44.181958 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j7nx\" (UniqueName: \"kubernetes.io/projected/4fdf0956-2383-4a6c-a5a2-c62c38854104-kube-api-access-7j7nx\") pod \"maas-api-revoke-sub-b4f5d22e-scnr5\" (UID: \"4fdf0956-2383-4a6c-a5a2-c62c38854104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5" Nov 02 18:26:44.182186 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:44.182092 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fdf0956-2383-4a6c-a5a2-c62c38854104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4f5d22e-scnr5\" (UID: \"4fdf0956-2383-4a6c-a5a2-c62c38854104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5" Nov 02 18:26:44.283341 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:44.283263 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fdf0956-2383-4a6c-a5a2-c62c38854104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4f5d22e-scnr5\" (UID: \"4fdf0956-2383-4a6c-a5a2-c62c38854104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5" Nov 02 18:26:44.283341 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:44.283297 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j7nx\" (UniqueName: \"kubernetes.io/projected/4fdf0956-2383-4a6c-a5a2-c62c38854104-kube-api-access-7j7nx\") pod \"maas-api-revoke-sub-b4f5d22e-scnr5\" (UID: \"4fdf0956-2383-4a6c-a5a2-c62c38854104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5" Nov 02 18:26:44.283525 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:44.283348 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fdf0956-2383-4a6c-a5a2-c62c38854104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4f5d22e-scnr5\" (UID: \"4fdf0956-2383-4a6c-a5a2-c62c38854104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5" Nov 02 18:26:44.284090 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:44.284067 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fdf0956-2383-4a6c-a5a2-c62c38854104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4f5d22e-scnr5\" (UID: \"4fdf0956-2383-4a6c-a5a2-c62c38854104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5" Nov 02 18:26:44.285659 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:44.285641 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fdf0956-2383-4a6c-a5a2-c62c38854104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4f5d22e-scnr5\" (UID: \"4fdf0956-2383-4a6c-a5a2-c62c38854104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5" Nov 02 18:26:44.306656 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:44.306622 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j7nx\" (UniqueName: \"kubernetes.io/projected/4fdf0956-2383-4a6c-a5a2-c62c38854104-kube-api-access-7j7nx\") pod \"maas-api-revoke-sub-b4f5d22e-scnr5\" (UID: \"4fdf0956-2383-4a6c-a5a2-c62c38854104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5" Nov 02 18:26:44.331313 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:44.331285 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5" Nov 02 18:26:44.633657 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:44.633630 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5"] Nov 02 18:26:44.635704 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:26:44.635662 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fdf0956_2383_4a6c_a5a2_c62c38854104.slice/crio-93f382e98fcf2bb4783177d32c66a72ada1bf728dff10f1135f2e521c2bab41e WatchSource:0}: Error finding container 93f382e98fcf2bb4783177d32c66a72ada1bf728dff10f1135f2e521c2bab41e: Status 404 returned error can't find the container with id 93f382e98fcf2bb4783177d32c66a72ada1bf728dff10f1135f2e521c2bab41e Nov 02 18:26:45.551191 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:45.551151 4860 generic.go:356] "Generic (PLEG): container finished" podID="4fdf0956-2383-4a6c-a5a2-c62c38854104" containerID="01378da824d6e6cfd1a351fe92a5eac294f9933e14bdd6bd4572ebdf2143566b" exitCode=0 Nov 02 18:26:45.551583 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:45.551191 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5" event={"ID":"4fdf0956-2383-4a6c-a5a2-c62c38854104","Type":"ContainerDied","Data":"01378da824d6e6cfd1a351fe92a5eac294f9933e14bdd6bd4572ebdf2143566b"} Nov 02 18:26:45.551583 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:45.551251 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5" event={"ID":"4fdf0956-2383-4a6c-a5a2-c62c38854104","Type":"ContainerStarted","Data":"93f382e98fcf2bb4783177d32c66a72ada1bf728dff10f1135f2e521c2bab41e"} Nov 02 18:26:46.700025 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:46.699998 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5" Nov 02 18:26:46.807240 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:46.807208 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fdf0956-2383-4a6c-a5a2-c62c38854104-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fdf0956-2383-4a6c-a5a2-c62c38854104-maas-api-openshift-ca\") pod \"4fdf0956-2383-4a6c-a5a2-c62c38854104\" (UID: \"4fdf0956-2383-4a6c-a5a2-c62c38854104\") " Nov 02 18:26:46.807411 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:46.807301 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fdf0956-2383-4a6c-a5a2-c62c38854104-kube-api-access-7j7nx\" (UniqueName: \"kubernetes.io/projected/4fdf0956-2383-4a6c-a5a2-c62c38854104-kube-api-access-7j7nx\") pod \"4fdf0956-2383-4a6c-a5a2-c62c38854104\" (UID: \"4fdf0956-2383-4a6c-a5a2-c62c38854104\") " Nov 02 18:26:46.807411 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:46.807336 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fdf0956-2383-4a6c-a5a2-c62c38854104-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fdf0956-2383-4a6c-a5a2-c62c38854104-maas-api-xks-ca\") pod \"4fdf0956-2383-4a6c-a5a2-c62c38854104\" (UID: \"4fdf0956-2383-4a6c-a5a2-c62c38854104\") " Nov 02 18:26:46.807625 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:46.807602 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fdf0956-2383-4a6c-a5a2-c62c38854104-maas-api-openshift-ca" pod "4fdf0956-2383-4a6c-a5a2-c62c38854104" (UID: "4fdf0956-2383-4a6c-a5a2-c62c38854104"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:26:46.809386 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:46.809329 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fdf0956-2383-4a6c-a5a2-c62c38854104-maas-api-xks-ca" pod "4fdf0956-2383-4a6c-a5a2-c62c38854104" (UID: "4fdf0956-2383-4a6c-a5a2-c62c38854104"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:26:46.809499 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:46.809409 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fdf0956-2383-4a6c-a5a2-c62c38854104-kube-api-access-7j7nx" pod "4fdf0956-2383-4a6c-a5a2-c62c38854104" (UID: "4fdf0956-2383-4a6c-a5a2-c62c38854104"). InnerVolumeSpecName "kube-api-access-7j7nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:26:46.908324 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:46.908291 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fdf0956-2383-4a6c-a5a2-c62c38854104-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:26:46.908324 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:46.908319 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j7nx\" (UniqueName: \"kubernetes.io/projected/4fdf0956-2383-4a6c-a5a2-c62c38854104-kube-api-access-7j7nx\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:26:46.908525 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:46.908338 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fdf0956-2383-4a6c-a5a2-c62c38854104-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:26:47.564342 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:47.564312 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5" Nov 02 18:26:47.564506 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:47.564316 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5" event={"ID":"4fdf0956-2383-4a6c-a5a2-c62c38854104","Type":"ContainerDied","Data":"93f382e98fcf2bb4783177d32c66a72ada1bf728dff10f1135f2e521c2bab41e"} Nov 02 18:26:47.564506 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:47.564438 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93f382e98fcf2bb4783177d32c66a72ada1bf728dff10f1135f2e521c2bab41e" Nov 02 18:26:51.776872 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:26:51.776807 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:26:55.181379 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:55.181346 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth"] Nov 02 18:26:55.181997 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:55.181782 4860 cpu_manager.go:414] "containerMap: removing container" podUID="4fdf0956-2383-4a6c-a5a2-c62c38854104" containerName="revoke-keys" Nov 02 18:26:55.181997 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:55.181793 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fdf0956-2383-4a6c-a5a2-c62c38854104" containerName="revoke-keys" Nov 02 18:26:55.181997 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:55.181943 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fdf0956-2383-4a6c-a5a2-c62c38854104" containerName="revoke-keys" Nov 02 18:26:55.185095 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:55.185077 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth" Nov 02 18:26:55.201094 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:55.201068 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:26:55.204293 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:55.204265 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth"] Nov 02 18:26:55.213015 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:55.212992 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\"" Nov 02 18:26:55.287463 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:55.287422 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r85lj\" (UniqueName: \"kubernetes.io/projected/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-kube-api-access-r85lj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth\" (UID: \"5cd8865c-8baf-42bb-8a9d-55add1da1f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth" Nov 02 18:26:55.287656 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:55.287465 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth\" (UID: \"5cd8865c-8baf-42bb-8a9d-55add1da1f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth" Nov 02 18:26:55.287656 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:55.287615 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth\" (UID: \"5cd8865c-8baf-42bb-8a9d-55add1da1f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth" Nov 02 18:26:55.388919 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:55.388878 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r85lj\" (UniqueName: \"kubernetes.io/projected/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-kube-api-access-r85lj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth\" (UID: \"5cd8865c-8baf-42bb-8a9d-55add1da1f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth" Nov 02 18:26:55.388919 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:55.388917 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth\" (UID: \"5cd8865c-8baf-42bb-8a9d-55add1da1f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth" Nov 02 18:26:55.389168 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:55.388956 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth\" (UID: \"5cd8865c-8baf-42bb-8a9d-55add1da1f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth" Nov 02 18:26:55.389605 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:55.389581 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth\" (UID: \"5cd8865c-8baf-42bb-8a9d-55add1da1f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth" Nov 02 18:26:55.391401 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:55.391377 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth\" (UID: \"5cd8865c-8baf-42bb-8a9d-55add1da1f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth" Nov 02 18:26:55.411715 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:55.411694 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r85lj\" (UniqueName: \"kubernetes.io/projected/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-kube-api-access-r85lj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth\" (UID: \"5cd8865c-8baf-42bb-8a9d-55add1da1f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth" Nov 02 18:26:55.497583 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:55.497511 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth" Nov 02 18:26:55.802896 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:55.802705 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth"] Nov 02 18:26:56.622782 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:56.622743 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth" event={"ID":"5cd8865c-8baf-42bb-8a9d-55add1da1f5f","Type":"ContainerStarted","Data":"e83f7dd374293dbd04360680e3269305c6d3b695338971f0c2b2880b51a72536"} Nov 02 18:26:56.623206 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:56.622797 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth" event={"ID":"5cd8865c-8baf-42bb-8a9d-55add1da1f5f","Type":"ContainerStarted","Data":"5a3529cb6635a27cb63404e9d35963d6d536dc5d8ee4a89b20909e23619af350"} Nov 02 18:26:56.664423 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:56.664373 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth" podStartSLOduration=1.6643576580000001 podStartE2EDuration="1.664357658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:26:56.662610714 +0000 UTC m=+3435.283970492" watchObservedRunningTime="2026-11-02 18:26:56.664357658 +0000 UTC m=+3435.285717433" Nov 02 18:26:57.629770 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:57.629733 4860 generic.go:356] "Generic (PLEG): container finished" podID="5cd8865c-8baf-42bb-8a9d-55add1da1f5f" containerID="e83f7dd374293dbd04360680e3269305c6d3b695338971f0c2b2880b51a72536" exitCode=0 Nov 02 18:26:57.630251 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:57.629816 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth" event={"ID":"5cd8865c-8baf-42bb-8a9d-55add1da1f5f","Type":"ContainerDied","Data":"e83f7dd374293dbd04360680e3269305c6d3b695338971f0c2b2880b51a72536"} Nov 02 18:26:58.776367 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:58.776344 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth" Nov 02 18:26:58.820649 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:58.820620 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-kube-api-access-r85lj\" (UniqueName: \"kubernetes.io/projected/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-kube-api-access-r85lj\") pod \"5cd8865c-8baf-42bb-8a9d-55add1da1f5f\" (UID: \"5cd8865c-8baf-42bb-8a9d-55add1da1f5f\") " Nov 02 18:26:58.820815 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:58.820670 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-maas-api-xks-ca\") pod \"5cd8865c-8baf-42bb-8a9d-55add1da1f5f\" (UID: \"5cd8865c-8baf-42bb-8a9d-55add1da1f5f\") " Nov 02 18:26:58.820815 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:58.820711 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-maas-api-openshift-ca\") pod \"5cd8865c-8baf-42bb-8a9d-55add1da1f5f\" (UID: \"5cd8865c-8baf-42bb-8a9d-55add1da1f5f\") " Nov 02 18:26:58.821162 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:58.821133 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-maas-api-openshift-ca" pod "5cd8865c-8baf-42bb-8a9d-55add1da1f5f" (UID: "5cd8865c-8baf-42bb-8a9d-55add1da1f5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:26:58.823135 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:58.823107 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-maas-api-xks-ca" pod "5cd8865c-8baf-42bb-8a9d-55add1da1f5f" (UID: "5cd8865c-8baf-42bb-8a9d-55add1da1f5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:26:58.823243 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:58.823169 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-kube-api-access-r85lj" pod "5cd8865c-8baf-42bb-8a9d-55add1da1f5f" (UID: "5cd8865c-8baf-42bb-8a9d-55add1da1f5f"). InnerVolumeSpecName "kube-api-access-r85lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:26:58.921571 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:58.921494 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-maas-api-openshift-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:26:58.921571 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:58.921524 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r85lj\" (UniqueName: \"kubernetes.io/projected/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-kube-api-access-r85lj\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:26:58.921571 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:58.921534 4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd8865c-8baf-42bb-8a9d-55add1da1f5f-maas-api-xks-ca\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:26:59.643097 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:59.643064 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth" Nov 02 18:26:59.643097 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:59.643086 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth" event={"ID":"5cd8865c-8baf-42bb-8a9d-55add1da1f5f","Type":"ContainerDied","Data":"5a3529cb6635a27cb63404e9d35963d6d536dc5d8ee4a89b20909e23619af350"} Nov 02 18:26:59.643360 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:59.643130 4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a3529cb6635a27cb63404e9d35963d6d536dc5d8ee4a89b20909e23619af350" Nov 02 18:26:59.828634 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:26:59.828598 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg"] Nov 02 18:27:08.755307 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:08.755269 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth"] Nov 02 18:27:08.767336 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:08.767301 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe6df-m2jth"] Nov 02 18:27:09.497956 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:09.497919 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cd8865c-8baf-42bb-8a9d-55add1da1f5f" path="/var/lib/kubelet/pods/5cd8865c-8baf-42bb-8a9d-55add1da1f5f/volumes" Nov 02 18:27:14.848647 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:14.848600 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" podUID="c4fd5039-5514-4e23-b080-f5495484504a" containerName="main" containerID="cri-o://b966b88bab7f2cf504bddae6d5fd053ab14074539d9333f002659549eb198db1" gracePeriod=45 Nov 02 18:27:15.100540 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.100476 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:27:15.184309 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.184267 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-dshm\") pod \"c4fd5039-5514-4e23-b080-f5495484504a\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " Nov 02 18:27:15.184500 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.184315 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-home\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-home\") pod \"c4fd5039-5514-4e23-b080-f5495484504a\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " Nov 02 18:27:15.184500 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.184342 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-model-cache\") pod \"c4fd5039-5514-4e23-b080-f5495484504a\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " Nov 02 18:27:15.184500 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.184369 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4fd5039-5514-4e23-b080-f5495484504a-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4fd5039-5514-4e23-b080-f5495484504a-tls-certs\") pod \"c4fd5039-5514-4e23-b080-f5495484504a\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " Nov 02 18:27:15.184500 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.184416 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-tmp-dir\") pod \"c4fd5039-5514-4e23-b080-f5495484504a\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " Nov 02 18:27:15.184500 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.184445 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fd5039-5514-4e23-b080-f5495484504a-kube-api-access-rdxwg\" (UniqueName: \"kubernetes.io/projected/c4fd5039-5514-4e23-b080-f5495484504a-kube-api-access-rdxwg\") pod \"c4fd5039-5514-4e23-b080-f5495484504a\" (UID: \"c4fd5039-5514-4e23-b080-f5495484504a\") " Nov 02 18:27:15.184775 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.184597 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-home" pod "c4fd5039-5514-4e23-b080-f5495484504a" (UID: "c4fd5039-5514-4e23-b080-f5495484504a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:15.184775 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.184693 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-model-cache" pod "c4fd5039-5514-4e23-b080-f5495484504a" (UID: "c4fd5039-5514-4e23-b080-f5495484504a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:15.184996 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.184881 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-tmp-dir" pod "c4fd5039-5514-4e23-b080-f5495484504a" (UID: "c4fd5039-5514-4e23-b080-f5495484504a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:15.185065 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.185023 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:27:15.185065 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.185041 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:27:15.185065 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.185053 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:27:15.187269 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.187235 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fd5039-5514-4e23-b080-f5495484504a-kube-api-access-rdxwg" pod "c4fd5039-5514-4e23-b080-f5495484504a" (UID: "c4fd5039-5514-4e23-b080-f5495484504a"). InnerVolumeSpecName "kube-api-access-rdxwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:27:15.187269 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.187235 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4fd5039-5514-4e23-b080-f5495484504a-tls-certs" pod "c4fd5039-5514-4e23-b080-f5495484504a" (UID: "c4fd5039-5514-4e23-b080-f5495484504a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:27:15.187470 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.187269 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-dshm" pod "c4fd5039-5514-4e23-b080-f5495484504a" (UID: "c4fd5039-5514-4e23-b080-f5495484504a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:15.286291 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.286255 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4fd5039-5514-4e23-b080-f5495484504a-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:27:15.286291 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.286284 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4fd5039-5514-4e23-b080-f5495484504a-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:27:15.286291 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.286296 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdxwg\" (UniqueName: \"kubernetes.io/projected/c4fd5039-5514-4e23-b080-f5495484504a-kube-api-access-rdxwg\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:27:15.739320 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.739277 4860 generic.go:356] "Generic (PLEG): container finished" podID="c4fd5039-5514-4e23-b080-f5495484504a" containerID="b966b88bab7f2cf504bddae6d5fd053ab14074539d9333f002659549eb198db1" exitCode=0 Nov 02 18:27:15.739320 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.739320 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" Nov 02 18:27:15.739589 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.739361 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" event={"ID":"c4fd5039-5514-4e23-b080-f5495484504a","Type":"ContainerDied","Data":"b966b88bab7f2cf504bddae6d5fd053ab14074539d9333f002659549eb198db1"} Nov 02 18:27:15.739589 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.739408 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg" event={"ID":"c4fd5039-5514-4e23-b080-f5495484504a","Type":"ContainerDied","Data":"1795fb4ffad71786aa32b71402c550d1a0125c22a225f03e361a991b27ee8f0f"} Nov 02 18:27:15.739589 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.739425 4860 scope.go:122] "RemoveContainer" containerID="b966b88bab7f2cf504bddae6d5fd053ab14074539d9333f002659549eb198db1" Nov 02 18:27:15.749189 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.749166 4860 scope.go:122] "RemoveContainer" containerID="b966b88bab7f2cf504bddae6d5fd053ab14074539d9333f002659549eb198db1" Nov 02 18:27:15.749480 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:27:15.749459 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b966b88bab7f2cf504bddae6d5fd053ab14074539d9333f002659549eb198db1\": container with ID starting with b966b88bab7f2cf504bddae6d5fd053ab14074539d9333f002659549eb198db1 not found: ID does not exist" containerID="b966b88bab7f2cf504bddae6d5fd053ab14074539d9333f002659549eb198db1" Nov 02 18:27:15.749566 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.749491 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b966b88bab7f2cf504bddae6d5fd053ab14074539d9333f002659549eb198db1"} err="failed to get container status \"b966b88bab7f2cf504bddae6d5fd053ab14074539d9333f002659549eb198db1\": rpc error: code = NotFound desc = could not find container \"b966b88bab7f2cf504bddae6d5fd053ab14074539d9333f002659549eb198db1\": container with ID starting with b966b88bab7f2cf504bddae6d5fd053ab14074539d9333f002659549eb198db1 not found: ID does not exist" Nov 02 18:27:15.793967 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.793878 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg"] Nov 02 18:27:15.805728 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:15.805697 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-dbe6df/simulator-single-main-dbe6df-kserve-6c79dc76df-8tbsg"] Nov 02 18:27:17.498389 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:17.498351 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4fd5039-5514-4e23-b080-f5495484504a" path="/var/lib/kubelet/pods/c4fd5039-5514-4e23-b080-f5495484504a/volumes" Nov 02 18:27:19.815831 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:19.815793 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t"] Nov 02 18:27:19.816309 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:19.816257 4860 cpu_manager.go:414] "containerMap: removing container" podUID="5cd8865c-8baf-42bb-8a9d-55add1da1f5f" containerName="revoke-keys" Nov 02 18:27:19.816309 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:19.816270 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd8865c-8baf-42bb-8a9d-55add1da1f5f" containerName="revoke-keys" Nov 02 18:27:19.816309 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:19.816286 4860 cpu_manager.go:414] "containerMap: removing container" podUID="c4fd5039-5514-4e23-b080-f5495484504a" containerName="main" Nov 02 18:27:19.816309 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:19.816291 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4fd5039-5514-4e23-b080-f5495484504a" containerName="main" Nov 02 18:27:19.816545 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:19.816359 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd8865c-8baf-42bb-8a9d-55add1da1f5f" containerName="revoke-keys" Nov 02 18:27:19.816545 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:19.816370 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4fd5039-5514-4e23-b080-f5495484504a" containerName="main" Nov 02 18:27:19.821060 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:19.821036 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:19.848704 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:19.848667 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 18:27:19.848704 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:19.848674 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f7th6\"" Nov 02 18:27:19.848996 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:19.848729 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 18:27:19.848996 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:19.848769 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-d04a7d07-kserve-self-signed-certs\"" Nov 02 18:27:19.852879 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:19.852834 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t"] Nov 02 18:27:19.930086 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:19.930051 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-dshm\") pod \"e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:19.930086 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:19.930098 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b44dc67b-e804-4bde-a62b-4f041e5e5056-tls-certs\") pod \"e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:19.930310 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:19.930183 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-home\") pod \"e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:19.930310 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:19.930255 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq9bv\" (UniqueName: \"kubernetes.io/projected/b44dc67b-e804-4bde-a62b-4f041e5e5056-kube-api-access-fq9bv\") pod \"e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:19.930381 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:19.930306 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-tmp-dir\") pod \"e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:19.930381 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:19.930363 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-model-cache\") pod \"e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:20.031137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:20.031083 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b44dc67b-e804-4bde-a62b-4f041e5e5056-tls-certs\") pod \"e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:20.031349 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:20.031152 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-home\") pod \"e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:20.031349 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:20.031229 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq9bv\" (UniqueName: \"kubernetes.io/projected/b44dc67b-e804-4bde-a62b-4f041e5e5056-kube-api-access-fq9bv\") pod \"e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:20.031349 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:20.031256 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-tmp-dir\") pod \"e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:20.031349 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:20.031297 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-model-cache\") pod \"e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:20.031587 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:20.031379 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-dshm\") pod \"e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:20.033269 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:20.033235 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-home\") pod \"e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:20.033755 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:20.033728 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-tmp-dir\") pod \"e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:20.034164 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:20.034144 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-model-cache\") pod \"e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:20.040683 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:20.034373 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-dshm\") pod \"e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:20.040683 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:20.036742 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b44dc67b-e804-4bde-a62b-4f041e5e5056-tls-certs\") pod \"e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:20.060291 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:20.060265 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq9bv\" (UniqueName: \"kubernetes.io/projected/b44dc67b-e804-4bde-a62b-4f041e5e5056-kube-api-access-fq9bv\") pod \"e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:20.140432 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:20.140346 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:20.440483 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:20.440452 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t"] Nov 02 18:27:20.442016 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:27:20.441987 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb44dc67b_e804_4bde_a62b_4f041e5e5056.slice/crio-d3b7b03225367e222843d55f7eaccd88e6825c06001e02c6321323425cfc8363 WatchSource:0}: Error finding container d3b7b03225367e222843d55f7eaccd88e6825c06001e02c6321323425cfc8363: Status 404 returned error can't find the container with id d3b7b03225367e222843d55f7eaccd88e6825c06001e02c6321323425cfc8363 Nov 02 18:27:20.772662 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:20.772565 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" event={"ID":"b44dc67b-e804-4bde-a62b-4f041e5e5056","Type":"ContainerStarted","Data":"8386a7bb9504d337b5930318e9ba32af2d43d10efb36aab0fbb5e3dd1bf7c3cf"} Nov 02 18:27:20.772662 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:20.772610 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" event={"ID":"b44dc67b-e804-4bde-a62b-4f041e5e5056","Type":"ContainerStarted","Data":"d3b7b03225367e222843d55f7eaccd88e6825c06001e02c6321323425cfc8363"} Nov 02 18:27:20.815232 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:20.815167 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" podStartSLOduration=1.8151524669999999 podStartE2EDuration="1.815152467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:20.813019806 +0000 UTC m=+3459.434379582" watchObservedRunningTime="2026-11-02 18:27:20.815152467 +0000 UTC m=+3459.436512244" Nov 02 18:27:21.140971 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:21.140936 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:21.769412 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:27:21.769379 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:27:30.141452 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:30.141415 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:30.153873 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:30.153836 4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:30.885540 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:30.885511 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:27:42.772872 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:42.772007 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6gl6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:27:42.772872 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:42.772077 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" podUID="48139446-b329-47dc-a692-9b9a89e26ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:27:51.769987 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:27:51.769946 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:27:57.773490 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:57.772964 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6gl6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:27:57.773490 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:27:57.773040 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" podUID="48139446-b329-47dc-a692-9b9a89e26ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:28:04.046384 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:04.046345 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp"] Nov 02 18:28:04.058262 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:04.058232 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908b634a-8hfcp"] Nov 02 18:28:04.904236 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:04.904200 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t"] Nov 02 18:28:05.498593 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:05.498561 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4846f6c7-b7c8-4ced-bb3b-3ef9888fac93" path="/var/lib/kubelet/pods/4846f6c7-b7c8-4ced-bb3b-3ef9888fac93/volumes" Nov 02 18:28:19.926222 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:19.926172 4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" podUID="b44dc67b-e804-4bde-a62b-4f041e5e5056" containerName="main" containerID="cri-o://8386a7bb9504d337b5930318e9ba32af2d43d10efb36aab0fbb5e3dd1bf7c3cf" gracePeriod=45 Nov 02 18:28:20.179175 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.179118 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:28:20.229149 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.229115 4860 generic.go:356] "Generic (PLEG): container finished" podID="b44dc67b-e804-4bde-a62b-4f041e5e5056" containerID="8386a7bb9504d337b5930318e9ba32af2d43d10efb36aab0fbb5e3dd1bf7c3cf" exitCode=0 Nov 02 18:28:20.229350 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.229157 4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" Nov 02 18:28:20.229350 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.229205 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" event={"ID":"b44dc67b-e804-4bde-a62b-4f041e5e5056","Type":"ContainerDied","Data":"8386a7bb9504d337b5930318e9ba32af2d43d10efb36aab0fbb5e3dd1bf7c3cf"} Nov 02 18:28:20.229350 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.229256 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t" event={"ID":"b44dc67b-e804-4bde-a62b-4f041e5e5056","Type":"ContainerDied","Data":"d3b7b03225367e222843d55f7eaccd88e6825c06001e02c6321323425cfc8363"} Nov 02 18:28:20.229350 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.229280 4860 scope.go:122] "RemoveContainer" containerID="8386a7bb9504d337b5930318e9ba32af2d43d10efb36aab0fbb5e3dd1bf7c3cf" Nov 02 18:28:20.239059 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.239034 4860 scope.go:122] "RemoveContainer" containerID="8386a7bb9504d337b5930318e9ba32af2d43d10efb36aab0fbb5e3dd1bf7c3cf" Nov 02 18:28:20.239379 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:28:20.239353 4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8386a7bb9504d337b5930318e9ba32af2d43d10efb36aab0fbb5e3dd1bf7c3cf\": container with ID starting with 8386a7bb9504d337b5930318e9ba32af2d43d10efb36aab0fbb5e3dd1bf7c3cf not found: ID does not exist" containerID="8386a7bb9504d337b5930318e9ba32af2d43d10efb36aab0fbb5e3dd1bf7c3cf" Nov 02 18:28:20.239471 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.239389 4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8386a7bb9504d337b5930318e9ba32af2d43d10efb36aab0fbb5e3dd1bf7c3cf"} err="failed to get container status \"8386a7bb9504d337b5930318e9ba32af2d43d10efb36aab0fbb5e3dd1bf7c3cf\": rpc error: code = NotFound desc = could not find container \"8386a7bb9504d337b5930318e9ba32af2d43d10efb36aab0fbb5e3dd1bf7c3cf\": container with ID starting with 8386a7bb9504d337b5930318e9ba32af2d43d10efb36aab0fbb5e3dd1bf7c3cf not found: ID does not exist" Nov 02 18:28:20.282665 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.282625 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-tmp-dir\") pod \"b44dc67b-e804-4bde-a62b-4f041e5e5056\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " Nov 02 18:28:20.282665 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.282670 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-model-cache\") pod \"b44dc67b-e804-4bde-a62b-4f041e5e5056\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " Nov 02 18:28:20.282953 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.282723 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-home\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-home\") pod \"b44dc67b-e804-4bde-a62b-4f041e5e5056\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " Nov 02 18:28:20.282953 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.282743 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b44dc67b-e804-4bde-a62b-4f041e5e5056-tls-certs\" (UniqueName: \"kubernetes.io/secret/b44dc67b-e804-4bde-a62b-4f041e5e5056-tls-certs\") pod \"b44dc67b-e804-4bde-a62b-4f041e5e5056\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " Nov 02 18:28:20.282953 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.282808 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44dc67b-e804-4bde-a62b-4f041e5e5056-kube-api-access-fq9bv\" (UniqueName: \"kubernetes.io/projected/b44dc67b-e804-4bde-a62b-4f041e5e5056-kube-api-access-fq9bv\") pod \"b44dc67b-e804-4bde-a62b-4f041e5e5056\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " Nov 02 18:28:20.282953 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.282826 4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-dshm\") pod \"b44dc67b-e804-4bde-a62b-4f041e5e5056\" (UID: \"b44dc67b-e804-4bde-a62b-4f041e5e5056\") " Nov 02 18:28:20.283194 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.282964 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-tmp-dir" pod "b44dc67b-e804-4bde-a62b-4f041e5e5056" (UID: "b44dc67b-e804-4bde-a62b-4f041e5e5056"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:20.283194 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.283060 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-model-cache" pod "b44dc67b-e804-4bde-a62b-4f041e5e5056" (UID: "b44dc67b-e804-4bde-a62b-4f041e5e5056"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:20.283194 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.283146 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-home" pod "b44dc67b-e804-4bde-a62b-4f041e5e5056" (UID: "b44dc67b-e804-4bde-a62b-4f041e5e5056"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:20.283309 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.283223 4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-tmp-dir\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:28:20.283309 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.283246 4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-model-cache\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:28:20.285193 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.285167 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44dc67b-e804-4bde-a62b-4f041e5e5056-kube-api-access-fq9bv" pod "b44dc67b-e804-4bde-a62b-4f041e5e5056" (UID: "b44dc67b-e804-4bde-a62b-4f041e5e5056"). InnerVolumeSpecName "kube-api-access-fq9bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:20.285193 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.285188 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-dshm" pod "b44dc67b-e804-4bde-a62b-4f041e5e5056" (UID: "b44dc67b-e804-4bde-a62b-4f041e5e5056"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:20.285339 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.285256 4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44dc67b-e804-4bde-a62b-4f041e5e5056-tls-certs" pod "b44dc67b-e804-4bde-a62b-4f041e5e5056" (UID: "b44dc67b-e804-4bde-a62b-4f041e5e5056"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:28:20.386881 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.385321 4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-home\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:28:20.386881 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.385361 4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b44dc67b-e804-4bde-a62b-4f041e5e5056-tls-certs\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:28:20.386881 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.385385 4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq9bv\" (UniqueName: \"kubernetes.io/projected/b44dc67b-e804-4bde-a62b-4f041e5e5056-kube-api-access-fq9bv\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:28:20.386881 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.385400 4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b44dc67b-e804-4bde-a62b-4f041e5e5056-dshm\") on node \"ip-10-0-136-153.ec2.internal\" DevicePath \"\"" Nov 02 18:28:20.574401 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.574367 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t"] Nov 02 18:28:20.578631 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:20.578601 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-d04a7d07-kserve-977f44f6f-86v8t"] Nov 02 18:28:21.497497 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:21.497451 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b44dc67b-e804-4bde-a62b-4f041e5e5056" path="/var/lib/kubelet/pods/b44dc67b-e804-4bde-a62b-4f041e5e5056/volumes" Nov 02 18:28:21.769900 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:28:21.769785 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:28:30.971868 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:30.971692 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-2pq2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:28:30.971868 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:30.971784 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-2pq2k" podUID="68d5947c-9808-42a0-8a29-55b3590819d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:28:51.770575 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:28:51.770533 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:28:57.421118 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.421033 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-64678b9774-68rhx"] Nov 02 18:28:57.421501 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.421445 4860 cpu_manager.go:414] "containerMap: removing container" podUID="b44dc67b-e804-4bde-a62b-4f041e5e5056" containerName="main" Nov 02 18:28:57.421501 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.421457 4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44dc67b-e804-4bde-a62b-4f041e5e5056" containerName="main" Nov 02 18:28:57.421579 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.421533 4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44dc67b-e804-4bde-a62b-4f041e5e5056" containerName="main" Nov 02 18:28:57.424718 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.424693 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64678b9774-68rhx" Nov 02 18:28:57.451897 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.451839 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 18:28:57.452152 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.452134 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 18:28:57.454539 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.454512 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gml5d\"" Nov 02 18:28:57.454539 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.454515 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-64678b9774-68rhx"] Nov 02 18:28:57.572314 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.572284 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37139446-3790-4bc8-88ce-0cfac8fc4a4c-metrics-certs\") pod \"maas-api-64678b9774-68rhx\" (UID: \"37139446-3790-4bc8-88ce-0cfac8fc4a4c\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-68rhx" Nov 02 18:28:57.572494 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.572334 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hm6k\" (UniqueName: \"kubernetes.io/projected/37139446-3790-4bc8-88ce-0cfac8fc4a4c-kube-api-access-4hm6k\") pod \"maas-api-64678b9774-68rhx\" (UID: \"37139446-3790-4bc8-88ce-0cfac8fc4a4c\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-68rhx" Nov 02 18:28:57.572494 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.572420 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37139446-3790-4bc8-88ce-0cfac8fc4a4c-maas-api-tls\") pod \"maas-api-64678b9774-68rhx\" (UID: \"37139446-3790-4bc8-88ce-0cfac8fc4a4c\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-68rhx" Nov 02 18:28:57.572591 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.572497 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37139446-3790-4bc8-88ce-0cfac8fc4a4c-openshift-service-ca\") pod \"maas-api-64678b9774-68rhx\" (UID: \"37139446-3790-4bc8-88ce-0cfac8fc4a4c\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-68rhx" Nov 02 18:28:57.673906 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.673810 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hm6k\" (UniqueName: \"kubernetes.io/projected/37139446-3790-4bc8-88ce-0cfac8fc4a4c-kube-api-access-4hm6k\") pod \"maas-api-64678b9774-68rhx\" (UID: \"37139446-3790-4bc8-88ce-0cfac8fc4a4c\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-68rhx" Nov 02 18:28:57.673906 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.673901 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37139446-3790-4bc8-88ce-0cfac8fc4a4c-maas-api-tls\") pod \"maas-api-64678b9774-68rhx\" (UID: \"37139446-3790-4bc8-88ce-0cfac8fc4a4c\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-68rhx" Nov 02 18:28:57.674137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.673944 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37139446-3790-4bc8-88ce-0cfac8fc4a4c-openshift-service-ca\") pod \"maas-api-64678b9774-68rhx\" (UID: \"37139446-3790-4bc8-88ce-0cfac8fc4a4c\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-68rhx" Nov 02 18:28:57.674137 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.673998 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37139446-3790-4bc8-88ce-0cfac8fc4a4c-metrics-certs\") pod \"maas-api-64678b9774-68rhx\" (UID: \"37139446-3790-4bc8-88ce-0cfac8fc4a4c\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-68rhx" Nov 02 18:28:57.674712 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.674684 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37139446-3790-4bc8-88ce-0cfac8fc4a4c-openshift-service-ca\") pod \"maas-api-64678b9774-68rhx\" (UID: \"37139446-3790-4bc8-88ce-0cfac8fc4a4c\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-68rhx" Nov 02 18:28:57.676483 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.676457 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37139446-3790-4bc8-88ce-0cfac8fc4a4c-maas-api-tls\") pod \"maas-api-64678b9774-68rhx\" (UID: \"37139446-3790-4bc8-88ce-0cfac8fc4a4c\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-68rhx" Nov 02 18:28:57.676599 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.676495 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37139446-3790-4bc8-88ce-0cfac8fc4a4c-metrics-certs\") pod \"maas-api-64678b9774-68rhx\" (UID: \"37139446-3790-4bc8-88ce-0cfac8fc4a4c\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-68rhx" Nov 02 18:28:57.696652 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.696621 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hm6k\" (UniqueName: \"kubernetes.io/projected/37139446-3790-4bc8-88ce-0cfac8fc4a4c-kube-api-access-4hm6k\") pod \"maas-api-64678b9774-68rhx\" (UID: \"37139446-3790-4bc8-88ce-0cfac8fc4a4c\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-68rhx" Nov 02 18:28:57.737971 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:57.737936 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64678b9774-68rhx" Nov 02 18:28:58.037101 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:58.037072 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-64678b9774-68rhx"] Nov 02 18:28:58.038616 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:28:58.038589 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37139446_3790_4bc8_88ce_0cfac8fc4a4c.slice/crio-6a4852451eecbdc033c15e6103141f032d414d265ade2de3dc348dba132446e0 WatchSource:0}: Error finding container 6a4852451eecbdc033c15e6103141f032d414d265ade2de3dc348dba132446e0: Status 404 returned error can't find the container with id 6a4852451eecbdc033c15e6103141f032d414d265ade2de3dc348dba132446e0 Nov 02 18:28:58.040022 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:58.040003 4860 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:28:58.504808 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:58.504769 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64678b9774-68rhx" event={"ID":"37139446-3790-4bc8-88ce-0cfac8fc4a4c","Type":"ContainerStarted","Data":"727bb9b30b94a3410f27e96c8a98e4429aea21c3cc541209ccf8dace6502cb0d"} Nov 02 18:28:58.505274 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:58.504821 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64678b9774-68rhx" event={"ID":"37139446-3790-4bc8-88ce-0cfac8fc4a4c","Type":"ContainerStarted","Data":"6a4852451eecbdc033c15e6103141f032d414d265ade2de3dc348dba132446e0"} Nov 02 18:28:58.505274 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:58.504864 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-64678b9774-68rhx" Nov 02 18:28:58.547470 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:28:58.547418 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-64678b9774-68rhx" podStartSLOduration=1.245721772 podStartE2EDuration="1.547405163s" totalImagesPullingTime="301.683391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:28:58.546118369 +0000 UTC m=+3557.167478150" watchObservedRunningTime="2026-11-02 18:28:58.547405163 +0000 UTC m=+3557.168764943" Nov 02 18:29:04.518094 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:29:04.518061 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-64678b9774-68rhx" Nov 02 18:29:09.071452 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:29:09.071418 4860 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/NetworkManager-dispatcher.service\": RecentStats: unable to find data in memory cache]" Nov 02 18:29:21.769889 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:29:21.769836 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:29:42.232415 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:29:42.232379 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 18:29:42.246217 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:29:42.246186 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 18:29:42.249482 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:29:42.249445 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 18:29:42.262258 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:29:42.262236 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 18:29:44.626010 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:29:44.625973 4860 scope.go:122] "RemoveContainer" containerID="ac9affff94fe6e60b60fbbcc9930c4a8c7e8bbb015d0b1cac646da28c4e6582d" Nov 02 18:29:44.635643 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:29:44.635618 4860 scope.go:122] "RemoveContainer" containerID="c21a80badc5e0ae32696841531bd872484ff7f1b551e2f1866de8bfd7f053d79" Nov 02 18:29:51.776009 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:29:51.775966 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:30:21.769180 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:30:21.769137 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:30:51.770061 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:30:51.769957 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:31:21.768873 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:31:21.768811 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:31:42.771051 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:31:42.770999 4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6gl6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:31:42.771739 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:31:42.771700 4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6gl6p" podUID="48139446-b329-47dc-a692-9b9a89e26ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:31:47.030800 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:31:47.030763 4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5"] Nov 02 18:31:47.043117 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:31:47.043091 4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4f5d22e-scnr5"] Nov 02 18:31:47.497166 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:31:47.497130 4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fdf0956-2383-4a6c-a5a2-c62c38854104" path="/var/lib/kubelet/pods/4fdf0956-2383-4a6c-a5a2-c62c38854104/volumes" Nov 02 18:31:51.769252 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:31:51.769204 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:32:21.769358 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:32:21.769272 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:32:44.681347 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:32:44.681316 4860 scope.go:122] "RemoveContainer" containerID="01378da824d6e6cfd1a351fe92a5eac294f9933e14bdd6bd4572ebdf2143566b" Nov 02 18:32:45.960102 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:32:45.960064 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-gt7zt_412157d1-0980-4139-bff7-466a1df1b0d3/manager/0.log" Nov 02 18:32:46.412208 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:32:46.412174 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fdn9v_e8ee3d1e-c5bb-4c46-977e-cb4b2d183266/manager/0.log" Nov 02 18:32:46.946881 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:32:46.946839 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-h8bxb_9e543ec4-db22-4e57-bad4-ef8b2e1920e7/server/0.log" Nov 02 18:32:47.153761 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:32:47.153729 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kqv5l_3a1cac0c-2637-47b7-a9c4-d81bca026068/manager/0.log" Nov 02 18:32:47.387385 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:32:47.387344 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-bm8dm_cd9cd220-880d-4fd7-91ba-cc7bb242867e/proxy/0.log" Nov 02 18:32:48.147943 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:32:48.147909 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-64678b9774-68rhx_37139446-3790-4bc8-88ce-0cfac8fc4a4c/maas-api/0.log" Nov 02 18:32:49.731976 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:32:49.731941 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-8zvgb_11899ead-b87a-4f5e-9b8a-f2325cf41709/postgres/0.log" Nov 02 18:32:51.511318 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:32:51.511286 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-sjclp_48bcace2-94a5-4c4a-b7e7-ee2cb72dedfc/manager/0.log" Nov 02 18:32:51.708619 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:32:51.708585 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-d6j7v_d82bf74d-62d0-415d-aa93-abce6ae44ba7/manager/0.log" Nov 02 18:32:51.769462 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:32:51.769379 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:32:52.561339 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:32:52.561297 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-thmg4_995df54a-11d6-4015-9c3a-431d142cfba0/manager/0.log" Nov 02 18:32:53.343015 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:32:53.342980 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67c76445c-gvxk7_ac56f1a6-ffd6-4421-b2b9-8af5694db8b9/istio-proxy/0.log" Nov 02 18:32:54.154682 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:32:54.154648 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-2pq2k_68d5947c-9808-42a0-8a29-55b3590819d1/istio-proxy/0.log" Nov 02 18:32:54.762958 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:32:54.762929 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-6gl6p_48139446-b329-47dc-a692-9b9a89e26ed7/istio-proxy/0.log" Nov 02 18:32:55.955060 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:32:55.955018 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-w4772_94044c36-9d36-41a2-801c-afb684fdfd3c/istio-proxy/0.log" Nov 02 18:32:56.163823 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:32:56.163790 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn_bc64162d-415e-48ce-bfe3-8238c020fe58/istio-proxy/0.log" Nov 02 18:32:56.740489 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:32:56.740454 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79c77b5cd5-c6svb_3d83bf78-e7b7-48bb-98be-8196e92fd320/router/0.log" Nov 02 18:33:09.346535 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:33:09.346497 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-gt7zt_412157d1-0980-4139-bff7-466a1df1b0d3/manager/0.log" Nov 02 18:33:09.756867 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:33:09.756764 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fdn9v_e8ee3d1e-c5bb-4c46-977e-cb4b2d183266/manager/0.log" Nov 02 18:33:10.198090 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:33:10.198057 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-h8bxb_9e543ec4-db22-4e57-bad4-ef8b2e1920e7/server/0.log" Nov 02 18:33:10.392271 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:33:10.392241 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kqv5l_3a1cac0c-2637-47b7-a9c4-d81bca026068/manager/0.log" Nov 02 18:33:10.616065 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:33:10.616029 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-bm8dm_cd9cd220-880d-4fd7-91ba-cc7bb242867e/proxy/0.log" Nov 02 18:33:11.164743 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:33:11.164711 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-64678b9774-68rhx_37139446-3790-4bc8-88ce-0cfac8fc4a4c/maas-api/0.log" Nov 02 18:33:12.537069 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:33:12.537034 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-8zvgb_11899ead-b87a-4f5e-9b8a-f2325cf41709/postgres/0.log" Nov 02 18:33:13.957062 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:33:13.957031 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-sjclp_48bcace2-94a5-4c4a-b7e7-ee2cb72dedfc/manager/0.log" Nov 02 18:33:14.139134 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:33:14.139096 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-d6j7v_d82bf74d-62d0-415d-aa93-abce6ae44ba7/manager/0.log" Nov 02 18:33:14.933252 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:33:14.933214 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-thmg4_995df54a-11d6-4015-9c3a-431d142cfba0/manager/0.log" Nov 02 18:33:15.490641 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:33:15.490604 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67c76445c-gvxk7_ac56f1a6-ffd6-4421-b2b9-8af5694db8b9/istio-proxy/0.log" Nov 02 18:33:16.298667 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:33:16.298620 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-2pq2k_68d5947c-9808-42a0-8a29-55b3590819d1/istio-proxy/0.log" Nov 02 18:33:16.893278 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:33:16.893243 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-6gl6p_48139446-b329-47dc-a692-9b9a89e26ed7/istio-proxy/0.log" Nov 02 18:33:18.081390 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:33:18.081346 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-w4772_94044c36-9d36-41a2-801c-afb684fdfd3c/istio-proxy/0.log" Nov 02 18:33:18.272699 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:33:18.272664 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn_bc64162d-415e-48ce-bfe3-8238c020fe58/istio-proxy/0.log" Nov 02 18:33:18.824405 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:33:18.824373 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79c77b5cd5-c6svb_3d83bf78-e7b7-48bb-98be-8196e92fd320/router/0.log" Nov 02 18:33:21.770191 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:33:21.770109 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:33:44.704176 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:33:44.704141 4860 scope.go:122] "RemoveContainer" containerID="e83f7dd374293dbd04360680e3269305c6d3b695338971f0c2b2880b51a72536" Nov 02 18:33:51.770744 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:33:51.770700 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:34:21.769695 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:34:21.769654 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:34:40.114699 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:40.114664 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dt9rz/must-gather-h8zx6"] Nov 02 18:34:40.118503 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:40.118486 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dt9rz/must-gather-h8zx6" Nov 02 18:34:40.143694 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:40.143663 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rtx8\" (UniqueName: \"kubernetes.io/projected/9f22c2c3-35e2-470a-bc5d-c897bc5cfdcc-kube-api-access-8rtx8\") pod \"must-gather-h8zx6\" (UID: \"9f22c2c3-35e2-470a-bc5d-c897bc5cfdcc\") " pod="openshift-must-gather-dt9rz/must-gather-h8zx6" Nov 02 18:34:40.143834 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:40.143727 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f22c2c3-35e2-470a-bc5d-c897bc5cfdcc-must-gather-output\") pod \"must-gather-h8zx6\" (UID: \"9f22c2c3-35e2-470a-bc5d-c897bc5cfdcc\") " pod="openshift-must-gather-dt9rz/must-gather-h8zx6" Nov 02 18:34:40.181291 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:40.181261 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dt9rz\"/\"openshift-service-ca.crt\"" Nov 02 18:34:40.185972 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:40.185954 4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dt9rz\"/\"kube-root-ca.crt\"" Nov 02 18:34:40.186051 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:40.186035 4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dt9rz\"/\"default-dockercfg-rxf8r\"" Nov 02 18:34:40.220139 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:40.220104 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dt9rz/must-gather-h8zx6"] Nov 02 18:34:40.244254 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:40.244219 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rtx8\" (UniqueName: \"kubernetes.io/projected/9f22c2c3-35e2-470a-bc5d-c897bc5cfdcc-kube-api-access-8rtx8\") pod \"must-gather-h8zx6\" (UID: \"9f22c2c3-35e2-470a-bc5d-c897bc5cfdcc\") " pod="openshift-must-gather-dt9rz/must-gather-h8zx6" Nov 02 18:34:40.244410 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:40.244279 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f22c2c3-35e2-470a-bc5d-c897bc5cfdcc-must-gather-output\") pod \"must-gather-h8zx6\" (UID: \"9f22c2c3-35e2-470a-bc5d-c897bc5cfdcc\") " pod="openshift-must-gather-dt9rz/must-gather-h8zx6" Nov 02 18:34:40.244613 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:40.244595 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f22c2c3-35e2-470a-bc5d-c897bc5cfdcc-must-gather-output\") pod \"must-gather-h8zx6\" (UID: \"9f22c2c3-35e2-470a-bc5d-c897bc5cfdcc\") " pod="openshift-must-gather-dt9rz/must-gather-h8zx6" Nov 02 18:34:40.268496 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:40.268453 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rtx8\" (UniqueName: \"kubernetes.io/projected/9f22c2c3-35e2-470a-bc5d-c897bc5cfdcc-kube-api-access-8rtx8\") pod \"must-gather-h8zx6\" (UID: \"9f22c2c3-35e2-470a-bc5d-c897bc5cfdcc\") " pod="openshift-must-gather-dt9rz/must-gather-h8zx6" Nov 02 18:34:40.432693 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:40.432608 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dt9rz/must-gather-h8zx6" Nov 02 18:34:40.735334 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:40.735304 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dt9rz/must-gather-h8zx6"] Nov 02 18:34:40.736421 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:34:40.736392 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f22c2c3_35e2_470a_bc5d_c897bc5cfdcc.slice/crio-61946f4ed195d8c0554d266984dfc5552556e84ffeb776f4162a78171ce26833 WatchSource:0}: Error finding container 61946f4ed195d8c0554d266984dfc5552556e84ffeb776f4162a78171ce26833: Status 404 returned error can't find the container with id 61946f4ed195d8c0554d266984dfc5552556e84ffeb776f4162a78171ce26833 Nov 02 18:34:40.738396 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:40.738369 4860 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:34:41.642159 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:41.642113 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dt9rz/must-gather-h8zx6" event={"ID":"9f22c2c3-35e2-470a-bc5d-c897bc5cfdcc","Type":"ContainerStarted","Data":"61946f4ed195d8c0554d266984dfc5552556e84ffeb776f4162a78171ce26833"} Nov 02 18:34:42.283030 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:42.282940 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 18:34:42.297428 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:42.297379 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 18:34:42.298264 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:42.298236 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnhtvr_4dec5066-dab8-480c-946e-81c8772eacf0/extract/0.log" Nov 02 18:34:42.319953 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:42.316776 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 18:34:42.650781 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:42.650728 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dt9rz/must-gather-h8zx6" event={"ID":"9f22c2c3-35e2-470a-bc5d-c897bc5cfdcc","Type":"ContainerStarted","Data":"1b433826e58aa9bb87f799cf453a3fb3fd90eaee3dced3c598a02519117fdcf3"} Nov 02 18:34:42.651216 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:42.650794 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dt9rz/must-gather-h8zx6" event={"ID":"9f22c2c3-35e2-470a-bc5d-c897bc5cfdcc","Type":"ContainerStarted","Data":"dd720fd39704817cc9af3bd595a9756ddd020817ad8895f60b7dac3183dab966"} Nov 02 18:34:42.694014 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:42.693956 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dt9rz/must-gather-h8zx6" podStartSLOduration=1.656815723 podStartE2EDuration="2.693940707s" totalImagesPullingTime="1.037124984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:34:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:34:42.691459627 +0000 UTC m=+3901.312819403" watchObservedRunningTime="2026-11-02 18:34:42.693940707 +0000 UTC m=+3901.315300500" Nov 02 18:34:45.274818 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:45.274784 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w8vvj_b4de14e5-995e-4171-bb61-9f70e777688e/global-pull-secret-syncer/0.log" Nov 02 18:34:45.467766 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:45.467734 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7jhzf_e083a23b-76fd-410d-9633-ece3abc240e1/connection-checker/0.log" Nov 02 18:34:45.614150 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:45.614115 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-hxpgv_fbbf8db7-5d42-47b6-b03f-dcd4b68f3211/connection-checker/0.log" Nov 02 18:34:45.754815 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:45.754664 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-nxnkz_46d2456f-62b2-4aa5-b172-1c28ab7615ac/connection-checker/0.log" Nov 02 18:34:46.294873 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:46.294246 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vs4r4_daf51d1a-a65f-4fad-8a82-966b8bac7ba4/konnectivity-agent/0.log" Nov 02 18:34:46.421081 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:46.415002 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-153.ec2.internal_e3f259cc4f9c47e64336e525d3250602/haproxy/0.log" Nov 02 18:34:51.799961 ip-10-0-136-153 kubenswrapper[4860]: E1102 18:34:51.799905 4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fbrw7" containerName="extractor" Nov 02 18:34:52.683884 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:52.683833 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-sjclp_48bcace2-94a5-4c4a-b7e7-ee2cb72dedfc/manager/0.log" Nov 02 18:34:52.816488 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:52.816114 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-d6j7v_d82bf74d-62d0-415d-aa93-abce6ae44ba7/manager/0.log" Nov 02 18:34:53.406646 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:53.406606 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-thmg4_995df54a-11d6-4015-9c3a-431d142cfba0/manager/0.log" Nov 02 18:34:55.023258 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.023221 4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9"] Nov 02 18:34:55.028149 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.028120 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" Nov 02 18:34:55.047345 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.047314 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9"] Nov 02 18:34:55.068339 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.068303 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nzj4\" (UniqueName: \"kubernetes.io/projected/d55f6cc5-ed08-46b6-b730-a2b8062667b9-kube-api-access-2nzj4\") pod \"perf-node-gather-daemonset-b4wx9\" (UID: \"d55f6cc5-ed08-46b6-b730-a2b8062667b9\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" Nov 02 18:34:55.068537 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.068349 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d55f6cc5-ed08-46b6-b730-a2b8062667b9-sys\") pod \"perf-node-gather-daemonset-b4wx9\" (UID: \"d55f6cc5-ed08-46b6-b730-a2b8062667b9\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" Nov 02 18:34:55.068537 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.068453 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d55f6cc5-ed08-46b6-b730-a2b8062667b9-proc\") pod \"perf-node-gather-daemonset-b4wx9\" (UID: \"d55f6cc5-ed08-46b6-b730-a2b8062667b9\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" Nov 02 18:34:55.068537 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.068527 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d55f6cc5-ed08-46b6-b730-a2b8062667b9-podres\") pod \"perf-node-gather-daemonset-b4wx9\" (UID: \"d55f6cc5-ed08-46b6-b730-a2b8062667b9\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" Nov 02 18:34:55.068724 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.068554 4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d55f6cc5-ed08-46b6-b730-a2b8062667b9-lib-modules\") pod \"perf-node-gather-daemonset-b4wx9\" (UID: \"d55f6cc5-ed08-46b6-b730-a2b8062667b9\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" Nov 02 18:34:55.169755 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.169720 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nzj4\" (UniqueName: \"kubernetes.io/projected/d55f6cc5-ed08-46b6-b730-a2b8062667b9-kube-api-access-2nzj4\") pod \"perf-node-gather-daemonset-b4wx9\" (UID: \"d55f6cc5-ed08-46b6-b730-a2b8062667b9\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" Nov 02 18:34:55.169943 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.169772 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d55f6cc5-ed08-46b6-b730-a2b8062667b9-sys\") pod \"perf-node-gather-daemonset-b4wx9\" (UID: \"d55f6cc5-ed08-46b6-b730-a2b8062667b9\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" Nov 02 18:34:55.169943 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.169830 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d55f6cc5-ed08-46b6-b730-a2b8062667b9-proc\") pod \"perf-node-gather-daemonset-b4wx9\" (UID: \"d55f6cc5-ed08-46b6-b730-a2b8062667b9\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" Nov 02 18:34:55.169943 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.169872 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d55f6cc5-ed08-46b6-b730-a2b8062667b9-sys\") pod \"perf-node-gather-daemonset-b4wx9\" (UID: \"d55f6cc5-ed08-46b6-b730-a2b8062667b9\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" Nov 02 18:34:55.169943 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.169926 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d55f6cc5-ed08-46b6-b730-a2b8062667b9-podres\") pod \"perf-node-gather-daemonset-b4wx9\" (UID: \"d55f6cc5-ed08-46b6-b730-a2b8062667b9\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" Nov 02 18:34:55.169943 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.169932 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d55f6cc5-ed08-46b6-b730-a2b8062667b9-proc\") pod \"perf-node-gather-daemonset-b4wx9\" (UID: \"d55f6cc5-ed08-46b6-b730-a2b8062667b9\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" Nov 02 18:34:55.170128 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.169948 4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d55f6cc5-ed08-46b6-b730-a2b8062667b9-lib-modules\") pod \"perf-node-gather-daemonset-b4wx9\" (UID: \"d55f6cc5-ed08-46b6-b730-a2b8062667b9\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" Nov 02 18:34:55.170128 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.169999 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d55f6cc5-ed08-46b6-b730-a2b8062667b9-lib-modules\") pod \"perf-node-gather-daemonset-b4wx9\" (UID: \"d55f6cc5-ed08-46b6-b730-a2b8062667b9\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" Nov 02 18:34:55.170128 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.170036 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d55f6cc5-ed08-46b6-b730-a2b8062667b9-podres\") pod \"perf-node-gather-daemonset-b4wx9\" (UID: \"d55f6cc5-ed08-46b6-b730-a2b8062667b9\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" Nov 02 18:34:55.193748 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.193711 4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nzj4\" (UniqueName: \"kubernetes.io/projected/d55f6cc5-ed08-46b6-b730-a2b8062667b9-kube-api-access-2nzj4\") pod \"perf-node-gather-daemonset-b4wx9\" (UID: \"d55f6cc5-ed08-46b6-b730-a2b8062667b9\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" Nov 02 18:34:55.345043 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.345008 4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" Nov 02 18:34:55.678777 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.677747 4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9"] Nov 02 18:34:55.679972 ip-10-0-136-153 kubenswrapper[4860]: W1102 18:34:55.679927 4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd55f6cc5_ed08_46b6_b730_a2b8062667b9.slice/crio-421664d68711f8764ca41715db03bb4ff325ee96a064bd2e7780fe0ac2358d36 WatchSource:0}: Error finding container 421664d68711f8764ca41715db03bb4ff325ee96a064bd2e7780fe0ac2358d36: Status 404 returned error can't find the container with id 421664d68711f8764ca41715db03bb4ff325ee96a064bd2e7780fe0ac2358d36 Nov 02 18:34:55.744829 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.744797 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-pthgk_42f9e736-411e-46a2-89d0-2a3870e2f95a/cluster-monitoring-operator/0.log" Nov 02 18:34:55.809201 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.809166 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" event={"ID":"d55f6cc5-ed08-46b6-b730-a2b8062667b9","Type":"ContainerStarted","Data":"3f02a97d314dac2c4ac27557bdd1b375c4ffa5346a2950810c83e83c117c53ea"} Nov 02 18:34:55.809386 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.809217 4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" event={"ID":"d55f6cc5-ed08-46b6-b730-a2b8062667b9","Type":"ContainerStarted","Data":"421664d68711f8764ca41715db03bb4ff325ee96a064bd2e7780fe0ac2358d36"} Nov 02 18:34:55.809386 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.809375 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" Nov 02 18:34:55.853155 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:55.853085 4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" podStartSLOduration=1.8530644920000001 podStartE2EDuration="1.853064492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 18:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:34:55.851112926 +0000 UTC m=+3914.472472698" watchObservedRunningTime="2026-11-02 18:34:55.853064492 +0000 UTC m=+3914.474424265" Nov 02 18:34:57.156379 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:57.156350 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xw25t_729c02eb-e200-4928-b789-b312d38306e2/node-exporter/0.log" Nov 02 18:34:57.272906 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:57.272876 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xw25t_729c02eb-e200-4928-b789-b312d38306e2/kube-rbac-proxy/0.log" Nov 02 18:34:57.391383 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:34:57.391350 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xw25t_729c02eb-e200-4928-b789-b312d38306e2/init-textfile/0.log" Nov 02 18:35:01.423233 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:35:01.423197 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-7scpf_72445bf5-a7a3-49dc-9b97-5589ae901032/networking-console-plugin/0.log" Nov 02 18:35:01.826563 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:35:01.826538 4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-b4wx9" Nov 02 18:35:02.467509 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:35:02.467475 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/2.log" Nov 02 18:35:02.477107 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:35:02.477081 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xqzqj_5f9e8821-40a8-4012-b66a-813e80d856f2/console-operator/3.log" Nov 02 18:35:04.798508 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:35:04.798477 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-dmr89_f7a7133f-8372-49a3-8194-15ed2159aa67/volume-data-source-validator/0.log" Nov 02 18:35:07.335716 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:35:07.335690 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w7zsl_a453427b-5025-4653-a930-ce2cf2f41cce/dns/0.log" Nov 02 18:35:07.450608 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:35:07.450574 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w7zsl_a453427b-5025-4653-a930-ce2cf2f41cce/kube-rbac-proxy/0.log" Nov 02 18:35:07.646722 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:35:07.646623 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67c76445c-gvxk7_ac56f1a6-ffd6-4421-b2b9-8af5694db8b9/istio-proxy/0.log" Nov 02 18:35:08.064451 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:35:08.064426 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s6s58_9368ef45-b865-4cdf-95a6-5088fc54277e/dns-node-resolver/0.log" Nov 02 18:35:09.453482 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:35:09.453445 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jxsgc_b44f96b4-5c43-40cc-9c89-63d4ec368e8b/node-ca/0.log" Nov 02 18:35:11.319030 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:35:11.318986 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67c76445c-gvxk7_ac56f1a6-ffd6-4421-b2b9-8af5694db8b9/istio-proxy/0.log" Nov 02 18:35:13.134777 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:35:13.134737 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-2pq2k_68d5947c-9808-42a0-8a29-55b3590819d1/istio-proxy/0.log" Nov 02 18:35:13.393544 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:35:13.393452 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-6gl6p_48139446-b329-47dc-a692-9b9a89e26ed7/istio-proxy/0.log" Nov 02 18:35:13.787501 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:35:13.787424 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-w4772_94044c36-9d36-41a2-801c-afb684fdfd3c/istio-proxy/0.log" Nov 02 18:35:13.848043 ip-10-0-136-153 kubenswrapper[4860]: I1102 18:35:13.847992 4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-b6bf76947-nzbbn_bc64162d-415e-48ce-bfe3-8238c020fe58/istio-proxy/0.log"